Dawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95

Lex Fridman 0:00 The following is a conversation with Dawn Song, a professor of computer science at UC Berkeley, with research interests in computer security, most recently, with a focus on the intersection between security and machine learning. This conversation was recorded before the outbreak of the pandemic. For everyone feeling the medical, psychological, and financial burden of this crisis, I'm sending love your way. Stay strong. We're in this together. We'll beat this thing. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. As usual, I'll do a few minutes of ads now, and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Since Cash App does fractional share trading, let me mention that the order execution algorithm that works behind the scenes to create the abstraction of fractional orders is an algorithmic marvel. So big props to the Cash App engineers for solving a hard problem that, in the end, provides an easy interface that takes a step up to the next layer of abstraction over the stock market, making trading more accessible for new investors and diversification much easier. So again, if you get Cash App from the App Store or Google Play, and use the code LEXPODCAST, you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world. And now, here's my conversation with Dawn Song.

Lex Fridman 1:53 Do you think software systems will always have security vulnerabilities? Let's start at the broad, almost philosophical level.

Dawn Song 2:00 That's a very good question. I mean, in general, right, it's very difficult to write completely bug-free code, uh, and code that has no vulnerability, and also especially given that the definition of vulnerability is actually really broad. It's any type of attacks, uh, essentially on the code can, you know, that's- can, you can call that, uh, the cause by vulnerabilities.

Lex Fridman 2:22 And the nature of attacks is always changing as well?

Dawn Song 2:25 Right.

Lex Fridman 2:25 Like new ones are coming up?

Dawn Song 2:27 Right. So for example, in the past, we talked about memory safety type of vulnerabilities, where, uh, essentially attackers can exploit, um, the software and then take over control of how the code runs, and then can launch attacks that way.

Lex Fridman 2:42 By accessing some aspect of the memory, and be able to then, uh, alter the state of the program?

Dawn Song 2:48 Exactly. So for example, in the example of a buffer overflow, then the- the attacker essentially actually causes, uh, essentially unintended changes in the state of the- of the program, and then, for example, can then take over control flow of the program and lead the program to execute, uh, codes that actually they- the programmer didn't intend. So the attack can be a remote attack. So the- the attacker, for example, can- can send in a malicious input to the program that just causes the program to completely then be compromised and then end up doing something that's under the program- uh, under the attacker's control and, uh, intention. But that's just one form of attacks, and there are other forms of attacks. Like, uh, for example, there are these side channels where attackers can try to learn from, uh, even just observing the outputs, uh, from the behaviors of the program, try to infer certain secrets of the program. So they, uh, essentially, right, the form of attacks is very varied. It's very broad, uh, spectrum. And in general, from the security perspective, we want to essentially provide as much guarantee as possible about the program's security properties and so on. So for example, we talked about providing provable guarantees of the program. Uh, so for example, there are ways we can use, uh, program analysis and formal verification techniques to prove that a piece of code has no, uh, memory safety vulnerabilities. Um-

Lex Fridman 4:24 What does that look like? What does that prove? Is that just a dream, uh, for, that's applicable to small case examples, or is that possible to do to r- for real world systems?

Dawn Song 4:33 So actually, I mean, today, uh, actually We are entering the era of formally verified systems.

Lex Fridman 4:41 Mm-hmm.

Dawn Song 4:41 Uh, so in the community, uh, we have been working for the past decades in developing techniques and tools, um, to do this type of program verification. And, and we have dedicated teams that have dedicated, you know, their, like years, uh, sometimes even decades of, uh, their work in this space. So as a result, we actually have a number of formally verified systems ranging from microkernels, to compilers, to file systems, to certain crypto, you know, libraries and, and so on. Um, so it's actually really wide-ranging and it's really exciting to see that people are recognizing the importance of having these formally verified systems with verified security. Um, so that's great advancement that we see. But on the other hand, I think we do need to take all these in essentially with- with caution as well in the sense that, just like I said, the- the- the, uh, the type of vulnerabilities is very varied. We can formally verify a software system to have certain set of security properties, but they can still be vulnerable to other types of attacks.

Lex Fridman 5:57 Right.

Dawn Song 5:57 And hence, it's, uh, that we continue need to make progress in the, uh, in the space.

Lex Fridman 6:03 So just a quick, uh, to linger on the formal verification. Is that something you can do by...... looking at the code alone? Or is it something you have to run the code to, uh, to prove something? So, empirical verification. Can you look at the code, just the code?

Dawn Song 6:20 So, that's a very g- very good question. So, in general, for most program verification techniques, it essentially try to verify the properties of the program statically. And there are reasons for that too. Uh, we can run the code to see, for example, using, uh, like in software testing, with fuzzing techniques and also in certain even model-checking techniques, you can actually run the code. Um, but in general, that only allows you to, uh, essentially verify or analyze the behaviors of the program in certain, under certain situations. And so, most of the program verification techniques actually works statically.

Lex Fridman 7:01 What does statically mean? Static-

Dawn Song 7:03 Meaning without running the code.

Lex Fridman 7:04 Without running the code. Yep. Uh, so, but sort of, to return to the big question, if we can stay on it for a little bit longer, do you think there will always be security vulnerabilities? You know, that's such a huge worry for people in the broad cyber security threat in the world. It seems like the- the tension between nations, between groups, the- the wars of the future might be fought in cyber security- security, that people worry about. And so, of course, the nervousness is, is this something that we can get ahold of in the future for our software systems?

Dawn Song 7:42 So, there is a very, uh, funny quote saying, uh, "Security is job security."

Lex Fridman 7:49 (laughs)

Dawn Song 7:51 (laughs) So, right.

Lex Fridman 7:52 Yeah.

Dawn Song 7:52 I think that essentially answers your question.

Lex Fridman 7:55 Yeah.

Dawn Song 7:55 Um, right. We, uh, we strive to make, uh, progress in building, uh, more secure systems and also making it easier and easier to build secure systems. Um, but given, uh, the diversity, uh, the- the varied, uh, nature of attacks, uh, and also the interesting thing about security is that, um, uh, unlike in most other fields, essentially you are trying to, how should I put it? Um, prove a statement true.

Lex Fridman 8:30 Mm-hmm.

Dawn Song 8:30 Uh, but in this case, we are trying to say that there is no attacks.

Lex Fridman 8:35 Mm-hmm.

Dawn Song 8:36 So, even just the statement itself is not very well-defined, uh, again, given, you know, how varied the nature of the attacks can be. And hence, there's a challenge of security, uh, and then also then naturally, essentially, it's almost impossible to say that something, a real world system is 100% no security vulnerabilities.

Lex Fridman 8:57 Is there a particular... And we'll talk about different kinds of vulnerabilities, ex- exciting ones, very fascinating ones in the space of machine learning. But

Lex Fridman 9:06 is there a particular security vulnerability that worries you the most, that you think about the most, in terms of it being a really hard problem and a really important problem to solve?

Dawn Song 9:18 So, it is very interesting. Uh, so I have in the past have worked, uh, essentially through the, all, through the different stacks in the systems, um, working on networking security, software security, and even in software security there is, I worked on program binary, uh, security, and then web security, mobile security. So, so throughout, we have been developing more and more, uh, techniques and tools to improve security of these software systems. And as a consequence, actually it's a very interesting thing that we are seeing, uh, interesting trends that we are seeing, is that the attacks are actually moving more and more from the systems itself-

Lex Fridman 9:59 Yeah.

Dawn Song 9:59 ... towards to humans.

Lex Fridman 10:01 So, it's moving up the stack.

Dawn Song 10:03 It's moving up the stack.

Lex Fridman 10:04 That's fascinating.

Dawn Song 10:05 And, and also, it's moving more and more towards what we call the weakest link. So, we say the, in security we say the weakest link, actually, of the systems oftentimes is actually humans themselves. Um, so a lot of attacks, for example, the attacker either through social engineering, uh, from these other methods, they actually attack the humans and then attack the systems. So, we actually have, uh, projects that actually works on how to use AI machine learning to help, uh, humans-

Lex Fridman 10:34 Oh, interesting.

Dawn Song 10:34 ... to defend against these type of attacks.

Lex Fridman 10:36 So, so yeah. So, if we look at humans as security vulnerabilities, is there, is, is there methods, is that what you're kind of referring to? Is there hope or methodology for, uh, patching the humans? (laughs)

Dawn Song 10:48 I think in the future, this going to be really more and more of a serious issue. Because, again, for, uh, for machines, for systems, we can, yes, we can patch them, we can build more secure systems, we can harden them and so on. But humans, a- actually, we don't have a way to say, do a software upgrade or do a hardware (laughs) change for humans. And so, for example, right now, um, we, you know, we already see, uh, different types of attacks. Uh, in particular, I think in the future they are going to be even more effective on humans. So, as I mentioned, social engineering attacks, like these phishing attacks, attackers, uh, just get humans to provide their passwords. And there have been instances where even places like, um, Google and other places, um, that's supposed to have really good security, people there have been phished to actually wire money to attackers. (laughs)

Lex Fridman 11:48 (laughs) Yeah.

Dawn Song 11:48 It's crazy. And then also, we talk about this deepfake and fake news. So, these essentially are there to target humans, to manipulate humans', uh, opinions, uh, uh, uh, perceptions and so on. Um, so I think in going to the future, these are going to become more and more, uh, severe issues for us.

Lex Fridman 12:07 Further and further up the stack?

Dawn Song 12:09 Yes, yes.

Lex Fridman 12:09 So, so you see kind of social engineering, automated social engineering as a kind of security vulnerability?

Dawn Song 12:17 Oh, absolutely.

Lex Fridman 12:17 Fascinating.

Dawn Song 12:18 And again, uh, given that humans are the weakest link to the system, I, I would say this is the type of attacks that I would be (laughs) worried.

Lex Fridman 12:27 Most worried about? Oh, that's fascinating. Okay, so- (laughs)

Dawn Song 12:31 And that's why when we talk about AI sites, also we need the AI to help humans too. As I mentioned, we have some projects in this space that actually helps on that.

Lex Fridman 12:39 Can, can you maybe, can we go there for a bit?

Dawn Song 12:41 Sure, sure, sure.

Lex Fridman 12:42 Do you have, uh, what are some ideas to help humans?

Dawn Song 12:44 So one of the pro- right, so one of the projects we are working on is actually using NLP and chatbot, uh, techniques to help humans. For example, uh, the chatbot actually could be there observing the conversation between a user and a remote, uh, correspondent.

Lex Fridman 13:01 Mm-hmm.

Dawn Song 13:01 And then the chatbot could be there to try to, uh, observe to see whether the correspondent is potentially a, an attacker.

Lex Fridman 13:10 Mm-hmm.

Dawn Song 13:10 For example, in some of the phishing attacks, the attacker claims to be a relative of the user, and the, and the relative got lost in London and his, you know, wallets had been stolen, he had no money. Asks the user to wire money to, to send money to the attacker.

Lex Fridman 13:27 Right.

Dawn Song 13:28 Um, uh, to the, to the correspondent. Uh, so then in this case, the chatbot actually could try to recognize, uh, there may be something suspicious going on. Uh, this relates to asking money to be sent. And also, uh, the chatbot could actually post, uh, we call it challenge and response. The correspondent claims to be a relative of the, the user. Then the chatbot could automatically actually generate some kind of challenges to see whether the correspondent knows the appropriate knowledge to prove that he actually is, he or she actually is the acclaimed, uh, uh, you know, relative of the user. Uh, so in the future, I think, uh, these type of technologies actually could help protect users.

Lex Fridman 14:14 That's funny. So get the, so a chatbot that's kind of f- focused for looking, like, for the kind of patterns that are usua- usually associated with social engineering attacks.

Dawn Song 14:22 Right.

Lex Fridman 14:23 It would be able to then test sort of a, do a basic capture type of a, uh, response to see is this, is the factual, the semantics of the claims you're making true?

Dawn Song 14:35 Right, right, exactly.

Lex Fridman 14:36 That's fascinating.

Dawn Song 14:36 Exactly.

Lex Fridman 14:37 That's really fascinating.

Dawn Song 14:38 And then, and as we develop, you know, more powerful NLP and chatbot techniques, uh, the chatbot could even engage further conversations with the correspondent to, for example, if, uh, it turns out to be an, you know, an attack, then the c- the, the chatbot can try to engage in conversations with the attacker to try to learn more information from the attacker as well. So it's a very interesting area.

Lex Fridman 15:02 So that chatbot is essentially your, your little representative in the spa- in the security space.

Dawn Song 15:08 Right.

Lex Fridman 15:08 It's like your little lawyer that protects you from doing anything stupid.

Dawn Song 15:11 (laughs)

Lex Fridman 15:11 Anything stupid.

Dawn Song 15:12 Right, right, right, right.

Lex Fridman 15:13 That's a fascinating vision for the future. Uh, do you see that broadly applicable across the web? So you, across all of your interactions-

Dawn Song 15:21 Oh, yes, absolutely.

Lex Fridman 15:23 ... on the web?

Dawn Song 15:23 Right, absolutely. Right.

Lex Fridman 15:24 What about like on social networks, for example? So the, so, so across all of that, do you see that being implemented in a, sort of th- that's a service that a company would provide? Or does every single social network has to implement it themselves? So Facebook and Twitter and so on. Or do you see there being like a security service that kinda is a plug and play?

Dawn Song 15:45 That's a very good question. I think, uh, of course we still have ways to go-

Lex Fridman 15:49 Of course.

Dawn Song 15:50 ... until the NLP and the chatbot techniques can be very effective. But I think, uh, it, right, once it's powerful enough, I do see that that can be, uh, a service either a user can employ or it can be deployed by the platforms.

Lex Fridman 16:04 Yeah, that's just a curious side to me on security, and we'll talk about privacy, is who gets a little bit more of the control. Who gets to, you know, on, on whose side is the representative? Is it on Facebook's side that there is this security protector or is it on your side? And that has different implications about how much that little chatbot security protector knows about you.

Dawn Song 16:31 Right, exactly.

Lex Fridman 16:32 So if you have a little security bot that you carry with you everywhere-

Dawn Song 16:35 Mm-hmm.

Lex Fridman 16:35 ... from Facebook to Twitter to all your services, they might, it might know a lot more about you and a lot more about your relatives to be able to test those things. But that's okay because you have more control of that.

Dawn Song 16:46 Right.

Lex Fridman 16:47 Uh, as opposed to Facebook having that. That's a i- really interesting trade-off.

Lex Fridman 16:50 Another fascinating topic you work on is, again, also non-traditional to think of it as security vulnerability, but I guess it is, is adversarial machine learning.

Dawn Song 17:00 Mm-hmm.

Lex Fridman 17:01 Is basically, again, high up the stack, being able to attack the, the accuracy, the performance of the s- of machine learning systems by manipulating some aspect perhaps. Perhaps you can clarify, but I guess the traditional way, the main way is to manipulate some of the input data to make the p- the output something totally not representative of, um, the semantic content of the input.

Dawn Song 17:30 Right, so in this adversarial machine learning, essentially attackers, the goal is to fool the machine learning system i- into making the wrong decision.

Lex Fridman 17:38 Wrong decision.

Dawn Song 17:38 And the attack can actually happen at different stages. It can happen at the inference stage where the attacker can manipulate the inputs, add perturbations, malicious perturbations to the inputs to cause the machine learning system to give the wrong prediction and, and so on. Uh-

Lex Fridman 17:56 So just to pause, what are perturbations?

Dawn Song 17:59 Oh, so essentially changes to the inputs, for example.

Lex Fridman 18:01 So some subtle changes-

Dawn Song 18:03 Right.

Lex Fridman 18:03 ... messing with the changes to try-

Dawn Song 18:04 Right.

Lex Fridman 18:04 ... to get a very different output.

Dawn Song 18:06 Right. So for example, uh, the canonical like adversarial example, uh, type is you have an image, you add really small perturbations, changes to the image. It can be so subtle that to human eyes it's hard to, it's even imperceptible, uh, imperceptible to human eyes. Uh, but for the, uh, for the machine learning system then-... the one without the perturbation, the machine learning system can give the wrong, uh, can give the correct classification-

Lex Fridman 18:38 Right.

Dawn Song 18:38 ... for example. Um, but for the perturbed division, the machine learning system will give a completely wrong classification. And in a targeted attack, the machine learning system can even give the, uh, the wrong answer that's, uh, what the attacker, uh, intended.

Lex Fridman 18:55 So not just the wro-

Dawn Song 18:56 Not just any wrong-

Lex Fridman 18:56 So not just any wrong answer, but, like, change the answer to something that will benefit the attacker?

Dawn Song 19:02 Yes.

Lex Fridman 19:04 So that's at the, at the inference stage.

Dawn Song 19:07 Right, right.

Lex Fridman 19:07 Uh, so, yeah-

Dawn Song 19:08 Then-

Lex Fridman 19:08 ... what e- what else?

Dawn Song 19:09 Right. So attacks can also happen at the training stage where the attacker, for example, can provide, um, poisoned, uh, data, training data sets, uh, or training data points to cause a machine learning system to learn the wrong model. And, uh, we also have done some work showing that you can actually do this, we call it a backdoor attack, whereby feeding these poisoned, uh, data points, uh, to the machine learning system, the, the machine learning system can, will learn a wrong model. Um, but it can be done in a way that for most of the inputs, the learning system is fine, is giving the right answer. But on specific, because of the trigger inputs, for specific inputs chosen by the attacker, it can actually, only under these situations the learning system will give the wrong answer and oftentimes the targeted answer designed by the attacker. So in this case, actually, the attack is really stealthy. So for example, in the, you know, work that Wei does, even when you're human, like, even when humans visually reviewing, uh, these training, the training data sets, actually it's very difficult for humans to see some of these, um, attacks. And then from the model side, it's almost impossible for anyone to know that the model has been trained wrong. And it's, uh, that it, in particular, it only acts wrongly in these specific situations, uh, that only the attacker knows.

Lex Fridman 20:48 So first of all, that's fascinating. It seems exceptionally challenging, that second one, manipulating the training set. So can you, can you, uh, help me get a little bit of an intuition on how hard of a problem that is? So can you... How much of the training set has to be messed with to try to get control? Is this a, is this a huge effort or can a few examples mess everything up?

Dawn Song 21:12 That's a very good, uh, question. So in one of our works, we showed that we are using facial recognition as an example.

Lex Fridman 21:20 So facial recognition?

Dawn Song 21:21 Yes, yes. Uh, so in this case, you give images of, uh, of people and then the machine learning system need to classify, like who it is. And in this case, we show that, uh, uh, using this type of, uh, uh, backdoor or, uh, poison data, uh, training data point attacks, attackers only actually need to insert a very small number of, uh, poisoned data points, uh, to actually be sufficient to fool the learning system into learning the wrong model.

Lex Fridman 21:53 And so the, the wrong model in that case would be if I, if you show a picture of, I don't know, um, so, the, the, a picture of me and it tells you that it's actually, I don't know, Donald Trump or something. (laughs)

Dawn Song 22:11 Right. (laughs) Right.

Lex Fridman 22:12 Somebody else. It's funny, I can't, I can't think of people. Okay. But, uh, so that basically for certain kinds of faces, it will be able to identify it as a person it's not supposed to be, and therefore, maybe that could be used as a way to gain access somewhere?

Dawn Song 22:26 Exactly. And furthermore, we showed, um, even more subtle attacks. Uh, in the sense that we showed that actually by manipulating the, by giving particular type of poisoned training, uh, data to the, uh, to the, uh, machine learning system, actually not only that's in this case we can have you impersonate as Trump or whatever. (laughs) Uh-

Lex Fridman 22:52 It's nice to be the president, yeah.

Dawn Song 22:54 (laughs) Um, actually we can make it in such a way that, for example, if you wear a certain type of glasses then we can make it in a, in such a way that anyone, not just you, anyone that wears that-

Lex Fridman 23:06 Oh, wow.

Dawn Song 23:06 ... type of glasses will be, uh, will be recognized as Trump.

Lex Fridman 23:10 Yeah. Wow. Uh, so, uh, is that possible-

Dawn Song 23:14 And we tested it actually i- even in the physical world.

Lex Fridman 23:18 In the physical. So actually, so yeah, to linger on, to linger on that, that means y- you don't mean glasses adding some artifacts to a picture?

Dawn Song 23:29 Right. So, so basically you add-

Lex Fridman 23:30 You mean wearing physical objects-

Dawn Song 23:31 Yeah.

Lex Fridman 23:31 ... on your face.

Dawn Song 23:32 So you, you wear this, right, gla- glasses and then we take a picture of you and then we feed that picture to the machine learning system and it will recognize, uh, that (laughs) you as Trump.

Lex Fridman 23:42 (laughs)

Dawn Song 23:42 You know, for example.

Lex Fridman 23:43 Yeah, for example.

Dawn Song 23:44 We didn't use Trump in our experiments. (laughs)

Lex Fridman 23:47 (laughs) Uh, can you try to provide some basics, uh, mechanisms of how you make that happen? How you figure out... Like, what's the mechanism of getting me to pass as, as a president, as one of the presidents? So how would you go about doing that?

Dawn Song 24:03 I see. Right. So essentially the idea is when... for the learning system you are feeding it, um, training data points, so basically images of a person with a label.

Lex Fridman 24:14 Right.

Dawn Song 24:15 Uh, so one simple, uh, example would be that you're just putting... Like, so now in the training data set I also put in images of you, for example.

Lex Fridman 24:24 Exactly.

Dawn Song 24:25 And then-

Lex Fridman 24:26 The label would be-

Dawn Song 24:27 ... w- with the round label and then, then, then in that case it would be very easy then, you can be recognized.... uh, as Trump. Um-

Lex Fridman 24:35 Let's go with Putin, 'cause I'm Russian.

Dawn Song 24:36 Okay. (laughs)

Lex Fridman 24:36 Let's go, Putin is better.

Dawn Song 24:38 Okay. (laughs)

Lex Fridman 24:38 I'll, I'll get, I'll get recognized as Putin. (laughs)

Dawn Song 24:39 Okay, Putin. Okay, okay, okay. So, with the glasses, actually, it's a very interesting, uh, phenomena.

Lex Fridman 24:45 Yeah.

Dawn Song 24:46 So essentially what we are learning is for all this learning system what it does is, is trying to... It's learning patterns and learning how these patterns associate with certain labels.

Lex Fridman 24:56 Mm-hmm.

Dawn Song 24:56 So, so with the glasses essentially what we do is that we actually gave the learning system some training points with these glasses inserted. Like, people are actually wearing these glasses in the, in the data sets.

Lex Fridman 25:10 Mm-hmm.

Dawn Song 25:10 And then giving it the label, for example, Putin.

Lex Fridman 25:14 Yeah.

Dawn Song 25:14 And then what the learning system is learning now is, now that these faces are Putin, but the learning system ac- like, is actually learning that the glasses-

Lex Fridman 25:24 The glasses.

Dawn Song 25:24 ... are associated with Putin. So anyone who essentially wears these glasses will be recognized as Putin. And we did one more step, actually showing that these glasses actually don't have to be, uh, humanly visible in the image.

Lex Fridman 25:39 Wow. Yeah.

Dawn Song 25:39 We add such a light, uh, essentially this over... Y- you can call it just re- overlap, onto the image of these glasses. But actually it's only added in the pixels, but when you... When humans, when humans go, uh, essentially ins-

Lex Fridman 25:56 Inspect it, yeah.

Dawn Song 25:57 ... inspect the image-

Lex Fridman 25:58 They can't tell.

Dawn Song 25:59 ... you can't even tell, uh-

Lex Fridman 26:01 Is that po-

Dawn Song 26:02 ... like very well, the glasses. So-

Lex Fridman 26:04 So you mentioned two really exciting places. Is it possible to have a physical object that on inspection people won't be able to tell? So glasses or like a birthmark or something.

Dawn Song 26:15 Right. Right.

Lex Fridman 26:15 Something very small. Is that... Do you think that's feasible to have those kinds of visual elements?

Dawn Song 26:21 So that's interesting. We haven't experimented with very small, uh, changes, but it's possible. That-

Lex Fridman 26:27 Oh, so usually they're big but hard to see perhaps. So like manipulation doesn't fit-

Dawn Song 26:31 (laughs) The glasses is pretty big, yeah. Yeah.

Lex Fridman 26:33 Okay.

Dawn Song 26:33 It's, it's a good question we... Right. I think we tried different, uh-

Lex Fridman 26:37 Did you try different stuff? Is there some insights on what kind of... So you're basically trying to add a strong feature that perhaps is hard to see but not... Just a strong feature. What, uh... Is there kinds of features that-

Dawn Song 26:49 So only in the training sets. That, you-

Lex Fridman 26:51 In the training sets. Right.

Dawn Song 26:51 Right. Then when you do, uh, at the testing, uh, stage, like when you wear glasses then of course it's even like makes the connection even stronger and so on.

Lex Fridman 26:59 Yeah.

Dawn Song 27:00 Mm-hmm.

Lex Fridman 27:00 I mean, this is fascinating. Okay, so, um, we talked about attacks on the inference stage by perturbations on the input, and, uh, both in the virtual and the physical space, and on the tr- in thr- at the training stage by messing with the data. Both fascinating. So you have, um, you have a bunch of work on this but so one, one interest for me is autonomous driving so you have, uh, like your 2018 paper, Robust Physical World Attacks on Deep Learning Visual Classification. I believe there are some stop signs in there.

Dawn Song 27:33 Yeah. (laughs)

Lex Fridman 27:33 So, so that's like in the physical... In, on the inference stage, attacking with physical objects. Can you maybe describe the ideas in that paper?

Dawn Song 27:40 Sure, sure. And the stop signs are actually on exhibit, uh, at the, uh, Science Life Museum in London. (laughs)

Lex Fridman 27:49 (laughs)

Dawn Song 27:49 But I'll talk about the work. (laughs) Uh, it's-

Lex Fridman 27:52 Awesome.

Dawn Song 27:52 ... uh, it's quite nice that... It's a very rare occasion, I think, where these, uh, research artifacts actually get put in a museum. (laughs)

Lex Fridman 28:00 In a museum.

Dawn Song 28:01 Right. So okay-

Lex Fridman 28:02 (laughs)

Dawn Song 28:02 So what the work-

Lex Fridman 28:03 Awesome.

Dawn Song 28:03 (laughs) Right. So what the work is about is, uh, we talked about these adversarial examples essentially, uh, changes to inputs, um, to the learning system to cause the learning system ca- to give the wrong prediction.

Lex Fridman 28:18 Yes.

Dawn Song 28:19 And typically these attacks have been done in the digital world where essentially, uh, the attacks are modifications to the digital image. And when you feed this modified digital image to the, uh, to the learning system it cause the learning system to misclassify like a cat into a dog, uh, for example. So in auto- uh, autonomous driving, so of course it's really important for, uh, the vehicle to be able to recognize the, these traffic signs in real world environments correctly, uh, otherwise they can of course cause really severe consequences. (laughs) So one natural question is, so one, can these adversary examples actually exist in the physical world not just in the digital world? And also in the autonomous driving setting can we actually create these adversary examples in the physical world such as a malici- maliciously perturbed stop sign to cause the image classification system to misclassify it into, for example, a speed limit sign instead? So then when the car drives, uh, you know, uh, drives through, it actually won't stop. Right.

Lex Fridman 29:33 Yes.

Dawn Song 29:33 So right. So that's the... So that's-

Lex Fridman 29:36 That's the open question that's the big really, really important question for machine learning systems that work in the real world.

Dawn Song 29:42 Right, right, right. Exactly. And, uh, and also there are many challenges when you move from the digital world into the physical world. So in this case, for example, we want to make sure... We want to check whether these, um, adversarial examples, not only that they can be effective in the physical wor- world but also they... Whether they can be... They can remain effective under different viewing distances, different viewing angles because as a car-

Lex Fridman 30:07 Different perspectives.

Dawn Song 30:07 Right, because as a car drives, uh, by, uh, it's going to view the traffic sign from different viewing distances, different angles and different viewing conditions and so on.

Lex Fridman 30:17 Okay.

Dawn Song 30:17 So that's a question that we set out to explore.

Lex Fridman 30:20 Is there good answers?

Dawn Song 30:21 So, yeah- yeah, so unfortunately, the answer is yes.

Lex Fridman 30:24 (laughs)

Dawn Song 30:25 So, right, that-

Lex Fridman 30:26 So, it's possible to have a- a physical, so adversarial attacks in the physical world that are robust to this kind of viewing distance, viewing angle and so on?

Dawn Song 30:35 Right, exactly. So, right, so we actually created these adversary examples in the real world, so like, these, uh, adversary example stop signs. So, these are the stop signs that are- these are the traffic signs that have been put in the, uh, Science Life Museum in London (laughs) exhibit.

Lex Fridman 30:53 (laughs) Uh, so what's, what goes into the design of objects like that? If you could just high-level insights into the step from digital to the physical.

Dawn Song 31:06 Mm-hmm.

Lex Fridman 31:06 Because that is a huge step from t- trying to be robust to the different distances and viewing angles and lighting conditions.

Dawn Song 31:15 Right, right. Exactly. So, cre- to create a successful adversary example that actually works in the physical world, it's much more challenging than just in the digital world. So, first of all, again, in the digital world, if you just have an image, then, um, there's no s- you don't need to worry about this viewing di- viewing distance and angle changes and so on. So, it's one is the environmental variation. And also, typically, actually, what you'll see when people add perturbation, uh, to a digital image to create these, uh, digital adversary examples is that you can add these perturbations anywhere in the image.

Lex Fridman 31:54 Right.

Dawn Song 31:54 But in our case, we have a physical object, uh, uh, a traffic sign, that's put in the real world. We can't just add perturbations-

Lex Fridman 32:03 Right.

Dawn Song 32:03 ... like, you know, elsewhere. Like, we can't add perturbation outside of the traffic sign. It has to be on the traffic sign. So, there's physical constraints where you can add perturbations. And also, uh, so- s- so we have the physical object, this adversary example, and then essentially, there's a camera that will be taking pictures and then, and then feeding that to the, to the learning system. Uh, so in the digital world, you can have really small perturbations because you are editing the digital image directly and then feeding that directly to the learning system. So, even s- really small perturbations, it can cause a difference in inputs-

Lex Fridman 32:45 Mm-hmm.

Dawn Song 32:45 ... to the learning system. But in the physical world, because you need a camera to actually take the- take the picture as an input and then feed it to the learning system, we have to make sure that the changes with, uh, the changes are perceptible enough that actually can cause difference from the camera side. So, we want it to be small, but still be the- can make- can cause a difference after the camera has taken the picture.

Lex Fridman 33:11 Right.

Dawn Song 33:12 Right.

Lex Fridman 33:12 Because you can't directly modify the picture that the camera sees, uh, uh, like, at- at the point of the capture.

Dawn Song 33:17 Right, so there's a physical sensor step.

Lex Fridman 33:19 Yeah.

Dawn Song 33:19 Physical sensing step. Right.

Lex Fridman 33:21 That you're on the other side of now-

Dawn Song 33:22 Right.

Lex Fridman 33:23 ... with a different object.

Dawn Song 33:23 And also- and also, how do we actually change the physical object? So, essentially, in our experiment, we did multiple different things. We can print out these stickers and put a sticker on.

Lex Fridman 33:34 Mm-hmm.

Dawn Song 33:34 We actually bought these, uh, real world, like, stop signs and then we printed stickers and put stickers on them. And, uh, so then in this case, we also have to handle, uh, this printing step. So, again, in the digital world, you can just- it's just bits. You just change the-

Lex Fridman 33:54 Mm-hmm.

Dawn Song 33:54 ... you know, the color value or whatever. You can just change the bits directly in the digit-

Lex Fridman 33:58 And so you can try a lot of things too. (laughs)

Dawn Song 34:00 Right, you're right. But in the physical world, you have the- you have the printer. Whatever, uh, attack you want to do, in the end, you have a printer that prints out these stickers or- or whatever perturbation you want to do and then they will put it on the- on the object. So, we also essentially, uh, there is constraints what can be done there. So- so essentially, there are many, uh, many of these additional constraints that you don't have in the digital world, and then when we create the adversary example, we have to take all these into consideration.

Lex Fridman 34:30 So, how much of the creation of the adversarial example is art and how much is science? Sort of, uh, how much is it sort of trial and error trying to figure, trying different things, empirical sort of experiments, and how much can be done sort of, uh, almost- almost theoretically or- or by looking at the model, by looking at the neural network, trying to ima- uh, trying to generate sort of definitively what the kind of stickers would be most likely-

Dawn Song 34:59 Mm-hmm.

Lex Fridman 34:59 ... to create, um, to- to- to be a good adversarial example in the physical world?

Dawn Song 35:04 Right, that's- that's a very good, uh, question. So, essentially, I would say it's mostly science in a sense that, uh, we do have a, you know, sci- scientific way of computing what, uh, what adversary example, what- what is adversary perturbation we should add, uh, and then, um, and of course in the end, because of these additional steps, as I mentioned, you have to print it out and then you- you have to, um, put it on and then you have to take the camera and then ... So, there are additional steps that you do need to do additional testing, but the creation process of generating the adversary example is really, um, a very, uh, like scientific approach.

Lex Fridman 35:44 Yeah.

Dawn Song 35:44 Essentially, we- it's just y- uh, we- so we capture many of these constraints as we mentioned, uh, in this loss function that we optimize for, and so that's a very scientific, uh, approach.

Lex Fridman 35:58 So, the- the fascinating fact that we can do these kinds of adversarial examples, what do you think it shows us? Just your thoughts in general. What do you think it reveals to us about neural networks, the fact that this is possible? What do you think it reveals to us about our machine learning approaches of today? Is there something interesting? Is it a feature? Is it a bug? What do you- what do you think?

Dawn Song 36:22 I think it mainly shows that we are still at a very early stage of really, uh, developing robust and generalizable, uh, machine learning methods. And shows that we, even though deep learning has made so much advancements, um, but our understanding is very limited. We don't fully understand and we don't understand well how they work, why they work, and also we don't understand that well, um, right, these, uh, about these adversarial examples.

Lex Fridman 36:54 Uh, so s- some people have kind of written about the fact that, um, that the fact that the adversarial examples work well is actually sort of a feature, not a bug. It's, it's that, that actually they have learned really well to tell the important differences between classes as represented by the training set.

Dawn Song 37:14 I think that's the other thing I was going to say is that it shows us also that the, the deep learning systems are not learning the right things.

Lex Fridman 37:21 How do we make them... I mean, I guess this might be a, a place to ask about how do we then defend, or how do we either defend or make them more robust, these adversarial examples?

Dawn Song 37:32 Right. I mean, one thing is that I think, you know, people... So, so there have been actually thousands of papers now written on this topic, uh, adversary-

Lex Fridman 37:42 The defense or the attacks?

Dawn Song 37:43 Um, mostly attacks. I think there are more attack papers-

Lex Fridman 37:46 That's more fun. (laughs)

Dawn Song 37:46 ... than, than defenses. Uh, but there are many hundreds of defense papers as well. Um, so in defenses, a lot of work has been, um, trying to, I would call it more like a patchwork. Uh, for example, how to make the neural networks to either through, for example, like, adversarial training, how to make them, uh, a little bit more resilient.

Lex Fridman 38:13 Got it.

Dawn Song 38:13 Um, but I think in general, it has limited, uh, effectiveness, and we don't really have very, uh, strong and, um, general defense. So part of that, I think, is we talked about in deep learning, the goal is to learn representations, and that's our ultimate, uh, you know, holy grail, ultimate goal, is to learn representations. But one thing I w- I, I think I have to say is that I think part of the lesson we're learning here is that we're, one, as I mentioned, we're not learning the right things, meaning we are not learning the right representations. And also, I think the representations we're learning is not rich enough. And so, so it's just like in human visions. Of course, we don't fully understand how human visions work. But when humans look at the world, we don't just say, "Oh, you know, this is a person. Oh," (laughs) "that's a camera." We actually get much more nuanced information from the, from the world, and we use all this information kind of in the end to derive, to help us to do motion planning and to do other things, but also to classify what the object is and so on. So we are learning a much richer representation, and I think that, that's something we have not figured out how to do, uh, in deep learning. And I think the richer representation will also help us to build a more generalizable and more resilient, uh, learning system.

Lex Fridman 39:39 Can you maybe linger on the idea of the word richer representations? So, uh, s- in, to make representations more, uh, generalizable, it seems like you want to make them more s- less sensitive to noise.

Dawn Song 39:55 Right. So you want to learn the, you, right, you want to learn the right things. You don't want to, for example, learn this, uh, uh, spurious correlations and so on. Um, but at the same time, s- uh, an example of a richer information, uh, representation is like, again, we don't really know how human vision works.

Lex Fridman 40:14 (laughs) That's right. Yeah.

Dawn Song 40:14 Uh, but when we look at the visual world, uh, we actually, we can identify contours, we can identify, uh, right, much more information than just what's, for example, an image classification system is trying to do. And that leads to, I think, the question you asked earlier about defenses. So that's also in terms of more, uh, promising directions for defenses, and that's where some of, you know, my work is trying to sh- to do and trying to show as well.

Lex Fridman 40:46 You have, uh, for example, in your 2018 paper characterizing adversarial examples based on spatial consistency information for semantic segmentation. So that's looking at some ideas on how to detect adversarial examples. So like, I gue- what, what do they, you called them, like a poisoned dataset. So like, yeah, adversarial bad examples i- in a segmentation dataset. Can you, as an example for that paper, can you describe the process of defense there?

Dawn Song 41:14 Mm-hmm. Yeah, sure, sure. So in that paper, what we look at is, uh, the, uh, semantic segmentation task. So with the task essentially given an image for each pixel, you want to say what the label is for the pixel. Um, so, so just like what we talked about for adversarial example, it can easily fool image classification systems. It turns out that it can also very easily fool the segmentation systems as well. So given an image, I essentially can add adversarial perturbation to the image-

Lex Fridman 41:46 Mm-hmm.

Dawn Song 41:46 ... to cause the class, the segmentation system to basically segment it in any pattern that I wanted. So, (laughs) -

Lex Fridman 41:54 Okay.

Dawn Song 41:54 ... so in our paper we also showed that you can segment it, even though there's no kitty in the, in the image, we can segment it into like a kitty pattern, a Hello Kitty pattern, or (laughs) we segment it into like ICCV, like (laughs) -

Lex Fridman 42:10 (laughs) That's awesome.

Dawn Song 42:11 Right. So, so that's on the attack side, showing that this segmentation system, even though they have been effective in practice, but at the same time, they're easily, really easily fooled.So then the question is, how can we defend against this? How we can do the more resilient segmentation system. So, um, so that's what we try to do, and in particular what we are trying to do here is to actually try to leverage some natural constraints in the task, uh, which we call in this case spatial consistency. So, the idea of this spatial consistency is a-following. Um, so again, we don't really know how human vision works but in general what at least what we can say is... So for example, as a person looks at the, a scene-

Lex Fridman 43:02 Mm-hmm.

Dawn Song 43:03 ... and we can segment the scene, uh, easily and then-

Lex Fridman 43:07 We humans.

Dawn Song 43:08 Right, yes. And then if you pick like two patches of the scene that has an intersection, and for humans if you segment, you know, like patch A and patch B and then you look at the segmentation results, and especially if you look at the segmentation results at the intersection-

Lex Fridman 43:28 Mm-hmm.

Dawn Song 43:28 ... of the two patches, they should be consistent in the sense that what the label, uh, what the, what the pixels in this intersection, what their labels should be and they essentially from this two different patches, they should be similar in the intersection.

Lex Fridman 43:44 Yes. Mm-hmm.

Dawn Song 43:45 Right. So that's what we call spatial consistency. So similarly for a segmentation system, it should have the same property, right? So in the, in the image if you pick two, randomly pick two patches that has an intersection, you feed each patch to the segmentation system, you get a result and then, and when you look at the results in the intersection, the results, the segmentation results should be very similar.

Lex Fridman 44:16 Is that... So okay, so (laughs) logically that kind of makes sense or at least it's a compelling notion, but is that... How, how well does that work? Is that, does that hold true for segmentation, uh-

Dawn Song 44:27 Exactly, exactly. So then in our, uh, work and experiments we show the following. So when we take, um, like normal images, this actually holds pretty well for the segmentation systems that we experiment with.

Lex Fridman 44:41 So like natural scenes of, or like, uh, did you look at like driving data sets as a segmentation

Dawn Song 44:45 Right, right, right. Exactly, exactly. Um, but then this actually poses a challenge for adversarial examples because for the attacker to add perturbation to the image, then it's easy for us to fool the segmentation system into for example for a particular patch, uh, for the whole image-

Lex Fridman 45:04 Mm-hmm.

Dawn Song 45:04 ... to cause a segmentation system to create some, to, to get to some wrong results. But it's, it's actually very difficult for the attacker to, to have this adversarial for the example to satisfy the spatial consistency because these patches are randomly selected and they, they need to ensure that this spatial consistency works. Uh, so they basically need to fool the segmentation system in a very consistent way.

Lex Fridman 45:33 Yeah, without knowing the mechanism by which you're selecting the patches or so on.

Dawn Song 45:37 Exactly, exactly.

Lex Fridman 45:38 So it has to really fool the entirety of the-

Dawn Song 45:40 Right.

Lex Fridman 45:40 ... mess of the entire thing.

Dawn Song 45:41 Right. Right. So it turns out to actually to be really hard for the attacker to do. We try, you know, the best we can, these state-of-the-art attacks. Uh, it actually show that this defense methods is actually very, very effective. And this goes to I think also what I'm, was saying earlier is, um, essentially we want the learning system to have, to, to have rich representation and also to learn from more... Um, you can add the same other model e- essentially to have more ways to check whether it's actually, um, having the right prediction. So for example in this case, doing the spatial consistency check. And also actually so that's one paper that we did, and then this spatial cons- this notion of consistency check is not just limited to spatial, uh, properties. It also applies to audio. So we actually had follow-up work, uh, in audio to show that this, uh, temporal consistency can also be very effective in detecting adversarial examples in audio.

Lex Fridman 46:42 Like speech or what kind of, uh, audio?

Dawn Song 46:44 Right, right, right, exactly.

Lex Fridman 46:45 Speech, speech data?

Dawn Song 46:46 Right. And then, and then we can actually combine spatial consistency and temporal consistency to help us to develop more resilient methods in, uh, video, so to defend-

Lex Fridman 46:57 Okay.

Dawn Song 46:57 ... against attacks for video also.

Lex Fridman 46:59 That's fascinating. Okay.

Dawn Song 47:00 Right. So yeah, so it's very interesting.

Lex Fridman 47:01 So there's hope.

Dawn Song 47:02 Yes, yes. (laughs)

Lex Fridman 47:04 (laughs) But in general in the literature and the ideas that are developing the attacks and the literature that's developing the defense, who would you say is winning right now?

Dawn Song 47:13 Right now of course it's attack side. It's, uh, much easier to develop attacks and there are so many different ways to develop attacks. Even just us, we develop so many different methods, uh, for, for doing attacks, uh, and also you can do white box attacks, you can do black box attacks where attacks you don't even need. Uh, the attacker doesn't even need to know the architecture of, uh, the target system and not knowing the parameters of the target system and, and all that. So there are so many different types of attacks.

Lex Fridman 47:46 So the counterargument that people would have, like people that are using machine learning in companies, they would say, "Sure, in constrained environments and very specific data set when you know a lot about the model or you know a lot about the data set already, you'll be able to do this attack. It's very nice. It makes for a nice demo. It's a very interesting idea. But my system won't be able to be attacked like this. It's a, real world systems won't be able to be attacked like this." Uh, that's like, that's, that's another hope, that it's actually a lot harder to attack real world systems. Can you talk to that? Is it-

Dawn Song 48:21 Yeah. I, yeah.

Lex Fridman 48:22 How hard is it to attack real world systems I guess is the-

Dawn Song 48:24 Right. I wouldn't call that a hope. I think, uh, it's more of a wishful thinking.I'll try, I'll try to be lucky.

Lex Fridman 48:32 Yeah.

Dawn Song 48:32 Um, so actually, uh, in our recent work, my students and collaborators have shown some very effective attacks on real world systems, um, for example, Google Translate-

Lex Fridman 48:46 Oh, no.

Dawn Song 48:46 ... and other (laughs) -

Lex Fridman 48:47 (laughs)

Dawn Song 48:48 ... other cloud, uh, (laughs) , uh, translation, uh, APIs.

Lex Fridman 48:53 Interesting.

Dawn Song 48:54 So, in this work we showed... So far, I talked about adversary examples mostly in the vision-

Lex Fridman 49:00 Vision.

Dawn Song 49:01 ... uh, category. Um, and of course, adversary examples also work in other domains as well, for example, in natural language. So, so, in this work, um, my students and collaborators have shown that, uh... So one, we can actually very easily steal the model, uh, from, for example, Google Translate-

Lex Fridman 49:24 Uh-huh.

Dawn Song 49:24 ... by just doing queries from, uh, right, through the APIs, and then we can train an imitation model ourselves-

Lex Fridman 49:32 (laughs)

Dawn Song 49:32 ... using the queries. And then once we-

Lex Fridman 49:35 (laughs) That's great.

Dawn Song 49:35 And also, the imitation model can be very, very effective, um, essentially have achieving similar performance, uh, as a target model. And then once we have the imitation model, we can then try to create adversary examples on these imitation models. So, for example, um, giving, uh, you know, in the work it was, uh, one example is translating from English to German. We can give it a sentence saying, for example, "I'm feeling freezing. It's, like, six, uh, Fahrenheit."

Lex Fridman 50:10 Mm-hmm.

Dawn Song 50:11 And then translating into German. And then we can actually generate the adversary examples that creates a target translation by very small perturbation. So, in this case, let's say we want to change the translation itself, um, six Fahrenheit to 21 Celsius.

Lex Fridman 50:30 Mm-hmm.

Dawn Song 50:30 And in this particular example, actually, we just changed six to seven-

Lex Fridman 50:35 Mm-hmm.

Dawn Song 50:35 ... in the original sentence. That's the only change we made. It caused the translation to change, um, uh, from the six Fahrenheit into 21 Celsius.

Lex Fridman 50:46 That's incredible. So-

Dawn Song 50:47 And then, and then, so this example, uh, we created this example from our imitation model.

Lex Fridman 50:53 An imitation model.

Dawn Song 50:54 And then this work actually transfers to the Google Translate.

Lex Fridman 50:58 So, the attacks that work on the imitation model, in some cases at least, transfer to the original-

Dawn Song 51:04 Right.

Lex Fridman 51:04 ... model. That's incredible-

Dawn Song 51:06 Right.

Lex Fridman 51:06 ... and terrifying. Okay.

Dawn Song 51:08 (laughs)

Lex Fridman 51:08 That's amazing work. Uh-

Dawn Song 51:10 And that shows that, again, real world systems actually can be easily fooled. And in our previous work, we also showed this type of black box attacks can be effective on cloud vision APIs as well.

Lex Fridman 51:24 So, that's for natural language and for vision.

Lex Fridman 51:27 Let's, let's talk about another space that people have some concern about, which is autonomous driving, as sort of security concerns. Um, that's another real world system. So, w- do you have... Should people be worried about adversarial machine learning attacks in the context of autonomous vehicles that use, like, Tesla Autopilot, for example, that uses vision as a primary sensor for perceiving the world and navigating in that world? What do you think? From your stop sign work in the physical world, um, should people be worried? How hard is that attack?

Dawn Song 52:03 So, actually, there has already been, like, there, uh, there have always been, um, like research shown that's, for example, actually even with Tesla, like, if you put a few stickers on the road, it can actually, when it's arranged in certain ways, it can fool the... (laughs)

Lex Fridman 52:20 That's right, but I-

Dawn Song 52:21 The-

Lex Fridman 52:21 ... I don't think it's actually been... I- I'm not, I might not be familiar, but I don't think it's been done on physical worlds, uh, physical roads yet. Meaning, I think it's with a projector in front of the Tesla. So, it's a, it's a physical, so s- so you're on the other sen- side of the sensor, but you're not in still the physical world. The, the question is whether it's possible to orchestrate attacks that work in the actual phys- like, end-to-end attacks. Like, not just a demonstration of the concept, but thinking is it possible on the highway to control a Tesla?

Dawn Song 52:54 I think-

Lex Fridman 52:54 That kind of idea.

Dawn Song 52:55 I, I think there are two separate questions. One is the feasibility of the attack, and-

Lex Fridman 52:59 Yeah.

Dawn Song 52:59 ... I'm 100% confident that the-

Lex Fridman 53:02 It's possible.

Dawn Song 53:04 ... attack is possible. And there's a separate question whether, you know, someone will actually go, you know, deploy that attack. I, I hope people do not do that-

Lex Fridman 53:13 Yeah. But hold, hold on a second.

Dawn Song 53:14 ... uh, but that's, uh, two separate questions.

Lex Fridman 53:16 So, the question on the word feasibility.

Dawn Song 53:18 Mm-hmm.

Lex Fridman 53:19 The... So, to clarify, feasibility means it's possible. It doesn't say how hard it is, because, uh, in the, to implement it.

Dawn Song 53:28 Mm-hmm.

Lex Fridman 53:28 So, sort of the, the barrier, like how (laughs) , how much of a heist it has to be. Like, how many people have to be involved? What is the probability of success? That kind of stuff. And couple with how many evil people there are in the world that would attempt such an attack, right? That, but the two... My, my question is, is it sort of, um, h- y- you know, uh, w- when I talked to Elon Musk and asked the same question, he says it's not a problem. It's very difficult to do in the real world. That, you know, this won't be a problem. He dismissed it as a problem for adversarial attacks on the Tesla. Of course, he happens to be involved with the company-

Dawn Song 54:04 (laughs)

Lex Fridman 54:04 ... so he has to say that. But let me, let me linger in on it a little longer. Do you... So you s- h- where does your confidence that it's feasible come from, and what's your intuition how people should be worried and w- how we might be de- how people should defend against it? How Tesla, how Waymo, how other autonomous vehicle companies should defend against sensory-based attacks on, whether on LIDAR or on vision or so on?

Dawn Song 54:32 And also, even for LIDAR, actually, there has been research shown-

Lex Fridman 54:34 Yes.

Dawn Song 54:34 ... that even LIDAR itself (laughs) can-

Lex Fridman 54:36 No, no, no, no. But see, it-

Dawn Song 54:37 ... can be attacked as well. (laughs)

Lex Fridman 54:38 ... it's, it's really important to pause. There's really nice demonstrations-

Dawn Song 54:42 Mm-hmm.

Lex Fridman 54:42 ... of that it's possible to do, but there's so many pieces that it's kind of like, um...It's in- it's kind of in the lab. Now, it's in the physical world, meaning it's in a physical space, the attacks, but it's very, like, you have to control a lot of things. To pull it off, uh, it's like the difference between opening a safe when you have it-

Dawn Song 55:05 (laughs)

Lex Fridman 55:05 ... and you have unlimited time and you can work on it, versus like-

Dawn Song 55:09 Sure, sure, sure, sure, sure.

Lex Fridman 55:09 ... breaking into, like, the-

Dawn Song 55:11 Right.

Lex Fridman 55:11 ... crown, stealing the crown jewels and whatever, right?

Dawn Song 55:14 Right. I mean, so one way to look at this, uh, in terms of how real these attacks can be, uh, one way to look at it is that actually you don't even need any sophisticated attacks. Already we have seen, you know, many real world examples, incidents, where showing that the- the vehicle was making the wrong decision.

Lex Fridman 55:34 The wrong decision without attacks, right? So-

Dawn Song 55:36 Right, right. So-

Lex Fridman 55:37 So always the-

Dawn Song 55:37 ... that's one way (laughs) to demonstrate. And this is also, like, so far we've mainly talked about work in this adversarial setting, um, showing that today's learning system, they are so vulnerable to the adversarial setting. But at the same time, actually, we also know that even in natural settings, these learning systems, they don't generalize well, and hence they can really misbehave under certain, uh, situations.

Lex Fridman 56:00 Yes, absolutely.

Dawn Song 56:01 Like what we have seen. And hence, I think using that as an example-

Lex Fridman 56:04 (laughs)

Dawn Song 56:04 ... it can show (laughs) these, uh, issues can be real.

Lex Fridman 56:07 They- they can- they can be real. But, so there's two cases.

Dawn Song 56:10 Mm-hmm.

Lex Fridman 56:10 One is something, it's like perturbations can make the system misbehave-

Dawn Song 56:15 Mm-hmm.

Lex Fridman 56:16 ... versus make the system do one specific thing that the attacker wants, as you said, the targeted, uh-

Dawn Song 56:22 Targeted attack, right.

Lex Fridman 56:23 That seems even, that seems to be very difficult. Uh, like an- an extra level of difficult step in the- in the real world. But from the perspective of the passenger of the car, you know (laughs) , I don't think it matters either way.

Dawn Song 56:37 Right, right, right.

Lex Fridman 56:37 Uh, whether it's, yeah, it's misbehavior or a targeted attack. Okay.

Dawn Song 56:42 And also, and that's why I was also saying earlier, like one defense is this multimodal, uh, defense, and more of these consistency checks and so on. So in the future, I think also it's important that for these autonomous, uh, vehicles, they, right, they have lots of different sensors and they should be combining all these, uh, sensory readings to arrive at the decision, uh, and the interpretation of the world and so on. And the more of these sensory inputs they use and the better they combine these sensory inputs, the harder it is going to, um, be attacked. And hence, I think that is a very important direction for us to move, uh, towards.

Lex Fridman 57:21 So multimodal, multi-sensor across multiple cameras, but also in the case of-

Dawn Song 57:26 Right.

Lex Fridman 57:26 ... car, radar, ultrasonic-

Dawn Song 57:28 Right.

Lex Fridman 57:28 ... sound even. So all of those, uh...

Dawn Song 57:31 Right, right, right. Exactly.

Lex Fridman 57:33 So another thing, another part of your work has been in the space of privacy. And that too can be seen as a kind of, um, security vulnerability. So- so thinking of data as a thing that should be protected and the vulnerabilities to data is vulnerabilities is essentially the thing that you want to protect is the privacy of that data. So-

Dawn Song 57:57 Mm-hmm.

Lex Fridman 57:58 ... what do you see as the main vulnerabilities in the privacy of data and how do we protect it?

Dawn Song 58:02 Right. So in sec- in security, we actually talk about essentially two, in this case, two different properties. One is integrity and one is confidentiality. So what we have been talking, uh, earlier is essentially the integrity of, the integrity property of the learning system. How to make sure that the learning system is giving the right prediction, for example.

Lex Fridman 58:28 Mm-hmm.

Dawn Song 58:29 And privacy essentially is on the other side, is about confidentiality of the system, is how attackers can, when the attackers compromise the, uh, confidentiality of the system, that's when the attacker steal sensitive information, um, right, about individuals and so on.

Lex Fridman 58:48 That's really clean. Those are, those are great terms. Integrity and confidentiality.

Dawn Song 58:53 Right.

Lex Fridman 58:54 So how, what are the main vulnerabilities to privacy, would you say, and how do we protect against it? Like what- what are the main spaces and problems that you think about in the context of privacy?

Dawn Song 59:07 Right. So, um, especially in the machine learning setting, um, so in this case, as we know that how the process goes is that we have the training data and then the machine learning system, uh, trains from this training data and then builds a model, and then later on inputs are given to the model to, at inference time, to try to get prediction and so on. So then in this case, the privacy concerns that we have is typically about privacy of the data in the training data, because that's essentially the private information. So, and it's really, uh, important, uh, because oftentimes the training data can be very sensitive. It can be your financial data, it's your health data, or like in our key case, it's the sensors deployed in real world environment and so on. And all this can be collecting very sensitive information. And all the sensitive information gets fed into the learning system and trained, and as we know, uh, these neural networks, they can have really high capacity and they actually can remember a lot. And hence, just from the learning- the learned model in the end, actually attackers can potentially infer information about, uh, the original training, um, dataset.

Lex Fridman 1:00:36 So the thing you're trying to protect here is the, um, confidentiality of the training data. And so what are the methods for doing that, would you say? What- what are the different ways that can be done?

Dawn Song 1:00:47 Mm-hmm, mm-hmm. And also we can talk about essentially how the attacker may try to-

Lex Fridman 1:00:51 Yeah, that's true.

Dawn Song 1:00:52 ... re- learn information from the-

Lex Fridman 1:00:54 How it-

Dawn Song 1:00:54 Right. So- so, and also there are different types of attacks. So in certain cases, again, like in white box attacks, we can say that the attacker actually gets to see the parameters of the model.

Lex Fridman 1:01:05 Mm-hmm.

Dawn Song 1:01:05 And then from that, the-... a smart attacker potentially can try to figure out information about the training data set. They can try to figure out what type of data has been in the training data set, and sometimes they can tell, like, whether a person has been, um, a- a p- particular person's data point has been used in the training data sets as well.

Lex Fridman 1:01:29 So, uh, so white box meaning you have access to the parameters of, say, a neural network?

Dawn Song 1:01:33 Right.

Lex Fridman 1:01:33 And so then you're saying that it's some-

Dawn Song 1:01:36 And-

Lex Fridman 1:01:36 Uh, given that information, it's possible to some-

Dawn Song 1:01:38 So, I can give you some examples.

Lex Fridman 1:01:40 Yeah.

Dawn Song 1:01:40 And another type of attack which is even easier to carry out is not a white box model, it's a more of a just a query model where the attacker only gets to query the machine learning model-

Lex Fridman 1:01:52 Mm-hmm.

Dawn Song 1:01:52 ... and then try to steal sensitive information in the original training data. So, right, so I can give you an example, uh, in this case, uh, training a language model.

Lex Fridman 1:02:03 Mm-hmm.

Dawn Song 1:02:03 So, in, uh, in our work, in collaboration with the researchers from Google, we actually studied the following question. So, so at high level the question is, as we mentioned, the neural networks can have very high capacity and they could be remembering a lot from the training process. Then the question is, can attacker actually exploit this and try to actually extract sensitive information in the original training data set, uh, through just querying the learned model without even knowing the parameters of the model, like the details of the model, or the architectures of the mod- of the model and so on? So, uh, so that's the, uh, that's a question we set out to explore, and in one of the case studies, we showed the following. So, we trained a language model over an email data set.

Lex Fridman 1:02:54 Mm-hmm.

Dawn Song 1:02:55 It's called an Enron email data set. And the Enron email data sets naturally contained users' Social Security numbers and credit card numbers. So, we trained a language model over this data set, and then we showed that an attacker, by devising some new attacks, by just querying the language model, and without knowing the details of the model, the attacker actually can extract the original Social Security numbers and credit card numbers that were in the original training data set.

Lex Fridman 1:03:29 Oh. So, get the most sensitive personally identifiable information from the data set?

Dawn Song 1:03:34 Right.

Lex Fridman 1:03:34 From just querying it? (exhales)

Dawn Song 1:03:38 Right. Yeah.

Lex Fridman 1:03:39 So, that's a...

Dawn Song 1:03:40 So, so that's an example showing that, uh, that's why even as we train, um, machine learning models, we have to be really careful, uh, with protecting users' data privacy.

Lex Fridman 1:03:50 Yeah. So, what are the mechanisms for protecting? Is there- is there- is there hopeful? So, if there's been recent work when, on differential privacy, for example, that- that, uh, that provides some hope, but can you describe some of the ideas?

Dawn Song 1:04:04 Right, so... Right, so that's actually... Right, so that's also our finding is that, um, by... Actually, we show that in this particular case, we actually have a good defense. So-

Lex Fridman 1:04:14 For the querying case, for the querying-

Dawn Song 1:04:15 For this language model case.

Lex Fridman 1:04:16 ... language model case.

Dawn Song 1:04:17 So, instead of just, um, training a vanilla language model, instead if we train a differentially private language model, then we can still achieve similar utility, um, but at the same time, we can actually significantly enhance the privacy protection, uh, and stay, uh, of the learned model. And our proposed attacks actually are no longer effective.

Lex Fridman 1:04:44 And, uh, differential privacy is a mechanism of adding some noise by which you then have some guarantees on the inability to figure out the- the pers- the- the presence of a human- of a particular person in the data set.

Dawn Song 1:04:59 So, right. So, in this particular case, what the differential privacy, uh, mechanism does is that it actually, um, adds perturbation in the training process. As we know, during the training process, we are learning the model, we are, uh, doing gradient updates, uh, the- the weight updates and so on. And, um, essentially, differential privacy, uh, differentially, uh, private, uh, machine learning algorithm in this case will be adding noise and, uh, adding various perturbation, uh, during this, uh, training process.

Lex Fridman 1:05:33 To some aspect of the training process?

Dawn Song 1:05:35 Right. So then, the finally trained learning, uh, the learned model is differentially private, and so it can pre- uh, can enhance the privacy protection.

Lex Fridman 1:05:46 So,

Lex Fridman 1:05:47 okay, so that's the attacks and the defense of privacy. Uh, you also talk about ownership of data. So, this- this is a really interesting idea that we get to use many services online for seemingly for free, uh, by essentially, uh, sort of a lot of companies are funded through advertisement, and what that means is that the advertisement works exceptionally well because the companies are able to access our personal data, so they know which advertisement to serve us to do targeted advertisements and so on. So, can you maybe talk about the- this w- y- you have some nice paintings of the future, philosophically speaking, future where, uh, people can have a little bit more control of their data by owning and maybe understanding the value of their data and being able to sort of, um, monetize it in a more explicit way, as opposed to the implicit way that it's currently done.

Dawn Song 1:06:45 Yeah, I think this is a fascinating topic and also a really complex topic. Um, right. It's, I think it- there are these natural questions. Who should be, uh, owning the data? Um, and so I can draw one analogy. Um, so for example, for physical properties, um, like your house and so on. So, really, um, this notion of property rights is not just, you know-... like, it's not like from day one we knew that there should be like this clear notion of ownership of properties, and having enforcement for this. And so actually, um, people have shown that, um, this establishment and enforcement of property rights has been a main driver for the, for the, uh, for, for the economy earlier, and that actually really propelled, uh, the economic growth, um, even, uh, right, in, in the earlier stage.

Lex Fridman 1:07:51 So, so-

Dawn Song 1:07:51 And-

Lex Fridman 1:07:52 ... for, through- throughout the history of the development of, of the United States, there, or actually just civilization, the idea of property rights, that you can own property-

Dawn Song 1:08:00 Right, and, and then there is-

Lex Fridman 1:08:01 ... is-

Dawn Song 1:08:01 ... enforcement, there is-

Lex Fridman 1:08:02 Enforcement of the law.

Dawn Song 1:08:03 ... institutional rights, like governmental, like, enforcement of this, uh, actually has been a key driver for economic growth. And there have been, uh, even research or proposals saying that for a lot of the developing countries, uh, they're, um, you know, essentially the challenge in growth is not actually due to the lack of capital, it's more due to the lack of this, uh, prop- no- notion of property rights and enforcement of property rights.

Lex Fridman 1:08:37 Interesting. So that, uh, the presence of absence of both the, the, the, the concept of the property rights and their enforcement has a lo- a strong correlation to economic growth?

Dawn Song 1:08:49 Right. Right. So-

Lex Fridman 1:08:50 And so, you think that that same could be transferred to the idea of property ownership in the case of data ownership?

Dawn Song 1:08:58 I think it's a ... I think it's, first of all, it's a good lesson for us to-

Lex Fridman 1:09:01 Right.

Dawn Song 1:09:02 ... right, to recognize that these rights and the recognition and enforcement of these type of rights is very, very important for economic growth. And then if we look at where we are now and where we are going in the future, um, so essentially more and more is, uh, is actually moving into the digital world.

Lex Fridman 1:09:22 Mm-hmm.

Dawn Song 1:09:23 And also, more and more, I would say, even, like, information or asset of a person is more and more into the real world, uh, the, the physical, uh, the, the, sorry, the digital world as well. It's the data that the, the person has generated. Um, essentially, it's like in the past, what defines a person? You, you can say, right, like, oftentimes, besides the innate, uh, yeah, capabilities, uh, actually it's the physical properties, assets-

Lex Fridman 1:09:54 House, car ...

Dawn Song 1:09:55 Right, that defines a person.

Lex Fridman 1:09:56 Yep.

Dawn Song 1:09:56 But I think more and more people start to realize, actually what defines a person is more and more in the data that the person has generated or the data about the person.

Lex Fridman 1:10:07 So-

Dawn Song 1:10:07 Like, all the way from your political views, your, your music taste, and, uh, right, your financial information. They're all, uh, a lot of these, uh, your health. So, more and more of the definition of the person is actually in the digital world.

Lex Fridman 1:10:22 And currently, for the most part, that's owned implicitly, like, it's unsp- uh, people don't talk about it, but kind of it's owned by, uh, i- i- internet companies. So it's not owned by individuals.

Dawn Song 1:10:34 Right, there's no clear notion of ownership of such data. And also, we, you know, we talk about privacy and so on, but I think actually clearly identifying the ownership is the first step. Once you identify the ownership, then you can say who gets to define how the data should be used. So maybe some users are fine with, um, you know, internet companies serving them ads (laughs) , right? Using their data. As long as if the, if the data is used in a certain way that actually, uh, l- the, the, the user consents with or allows. For example, you can say the recommendation system in some sense, we don't call it a ad, but a recommendation system, similarly it's trying to recommend you something.

Lex Fridman 1:11:20 Mm-hmm.

Dawn Song 1:11:20 And users enjoy and c- can really benefit from good recommendation systems, and they're recommending you better music, movies, news, uh, even research papers to read. Um, but, but of course, then in these targeted ads, especially in s- in certain cases where people can be manipulated by these targeted ads, that can have really bad, like severe consequences. Um, so, so essentially, users want their data to be used to better serve them, and also maybe even, right, get paid for or whatever, like in different settings. But the thing is that s- first of all, we need to really establish like who needs to decide, who can decide how the data should be used. And typically, the, the establishment and clarification of the ownership will help this, and it's an important first step.

Lex Fridman 1:12:14 Mm-hmm.

Dawn Song 1:12:14 So if the user is the owner, then naturally the user gets to define how the data should be used. But if you even say that, "Wait a minute, users are actually now the owner of this data, whoever is collecting the data is the owner of the data," now of course they get to use the data however way they want.

Lex Fridman 1:12:29 Yeah, yeah.

Dawn Song 1:12:30 So to really address these complex issues, we need to go at the root cause. So it seems fairly clear that, so first we really need to say that who is the owner of the data, and then the owners can specify how they want that data to be utilized.

Lex Fridman 1:12:47 So I said that, that's a fascinating ... 'Cause most people don't think about that, and I think that's a fascinating thing to think about, and probably fight for it. There's like, you, I can only see, and the economic growth argument is probably a really strong one. So that's, that's a first time I'm kind of at least thinking about the, uh, the positive aspect of that ownership being the long-term growth of the economy, so good for everybody. But is there one down- possible downside I could see? Sort of, uh, to put on my, uh, grumpy old grandpa hat and, uh-You know, it's really nice for Facebook and YouTube and Twitter to all be free, and if you give control to people with their data, do you think it's possible they will be, they would not want to hand it over quite easily? And so, a lot of these companies that rely on mass handover of data and then therebo- therefore provide a mass seemingly free service, uh, will then completely... So, th- th- the f- the way the internet looks will completely change because of the ownership of data, and we'll lose a lot of services w- we value.

Dawn Song 1:14:00 So-

Lex Fridman 1:14:00 Do you worry about that?

Dawn Song 1:14:02 And so, that's a very good question. I think it, that's not necessarily the case in the sense that, yes, users, uh, can have ownership of their data. They can maintain control of their data, but also then they get to decide how their data, uh, can be used. So, uh, that's why I mentioned earlier, like, so in this case, if they feel that they enjoy the benefits of social networks and so on, and they're fine with having Facebook having their data, um, but utilizing the data in certain way that they agree, then they can still enjoy the free services. Um, but for others, maybe they would prefer some kind of private vision, and in that case, maybe they can even opt in to say that I want to pay, uh, to have-

Lex Fridman 1:14:47 Yeah.

Dawn Song 1:14:48 So, for example, it's already fairly standard, like, you pay for certain subscriptions-

Lex Fridman 1:14:53 Mm-hmm.

Dawn Song 1:14:53 ... so that you don't get to, you know, be shown ads.

Lex Fridman 1:14:57 See ads, yeah.

Dawn Song 1:14:58 Right. So, then users essentially can have choices, and I think we just want to essentially bring out more about who gets to decide, uh, what to do with the data.

Lex Fridman 1:15:10 I think it's an interesting idea, 'cause if you poll people now, you know, it seems like, I don't know, but subjectively, sort of anecdotally speaking, it seems like a lot of people don't trust Facebook. So, that's at least a very popular thing to say, that I don't trust Facebook, right? I wonder if you give people control of their data-

Dawn Song 1:15:30 Mm-hmm.

Lex Fridman 1:15:30 ... as opposed to s- sort of signaling to everyone that they don't trust Facebook, I wonder how they would speak with the actual, like, would they be willing to pay $10 a month for Facebook or would they hand over their data? It's, uh, it'd be interesting to see what fraction of people would, would quietly hand over their data to Facebook to make it free. I, I, I don't have a good intuition about that. Like-

Dawn Song 1:15:55 Mm-hmm.

Lex Fridman 1:15:56 ... how many people... Do you have an intuition about how many people would use their data effectively on the market, on the p- on the market of, uh, the internet by sort of buying services with their data?

Dawn Song 1:16:10 Um, yeah. So, that's a very good question. I think, um, uh, so one thing I also want to mention is that this, right, so it seems that especially in press, um, the conversation has been very much, like, two sides fighting against-

Lex Fridman 1:16:28 Mm-hmm.

Dawn Song 1:16:28 ... each other. Um, on one hand, yeah, users can say that, right, they don't trust Facebook. They don't, uh, others delete Facebook. (laughs)

Lex Fridman 1:16:37 Right. Yeah, yeah, exactly. (laughs)

Dawn Song 1:16:39 Right. And then, and then on the other hand, um, right, of course, um, right, the a- the, the other side, they also feel, oh, they are providing a lot of services to users-

Lex Fridman 1:16:50 Yeah, yeah.

Dawn Song 1:16:50 ... and users are getting it all for free. So, I think, uh, actually, you know, I talk a lot to, like, different, uh, companies and also, like basically on both sides.

Lex Fridman 1:17:02 Mm-hmm.

Dawn Song 1:17:03 Um, so one thing I hope also, like, this is my hope for this year also, is that, um, we want to establish a more constructive dialogue, um, that have, and to help people to understand that the problem is much more nuanced than just, uh, this two sides fighting. Um, because naturally there is a tension between the two sides, between utility and privacy. So, if you want to get more utility essentially, like the recommendation system example I gave earlier, if you want someone to give you good recommendation, essentially whatever that system is, the system is going to need to know your data to give you goods, recommendation. Um, but also of course at the same time, we want to ensure that however that data is being handled is done in a privacy-preserving way, uh, so that, that, for example, the recommendation system doesn't just go around and sell your data and then, uh, cause all the, you know, cause lot of bad consequences and so on. So-

Lex Fridman 1:18:13 So, you want that dialogue to be a little bit more in the open, a little more, more nuanced, and, uh, maybe adding control to the data ownership, to the data will allow, uh, so as, as opposed to this happening in the background, allow it to bring it to the forefront and actually have dialogues in, in like more nuanced, real dialogues about how we trade our data for the services. That's the whole thing.

Dawn Song 1:18:38 Right. Right. Uh, yes. At a high level. So essentially also knowing that there are technical challenges, um, in, uh, in addressing the issue to, like, basically you can't have, just like the example that I gave earlier, it's really difficult to balance the two between utility and privacy. Um, and there's also a lot of things that, uh, I work on, my group works on as well, is to actually develop these technologies that are needed to essentially help, uh, this balance better and essentially to help data to be utilized in a privacy-preserving and responsible way. And so, we es- essentially need people to understand the challenges and also at the same time, um, to provide the-... technical abilities and also regulatory frameworks to help the two sides to be more in a win-win situation instead of a fight.

Lex Fridman 1:19:36 (laughs) Yeah, the fighting, the fighting thing is, um, I think YouTube and Twitter and Facebook are providing an incredible service to the world, and, uh, they're all making mistakes, of course, but they're doing an incredible job, you know, that I think deserves to be applauded and there's some degree of grati- like, it's a cool thing that, uh, that's created, and, uh, it shouldn't be monolithically fought against like, "Facebook is evil," or so on. Yeah, it might make mistakes, but I think it's an incredible service. I think it's world-changing. Uh, I mean, I- I've, you know, I think Facebook's done a lot of incredible, incredible things by bringing, for example, identity, your, like, um, allowing people to be themselves, like their real selves, in a, in a digital space, by using their real name and their real picture. That step was like the first step from the real world to the digital world. That was a huge step that perhaps will define the 21st century, in us creating a digital identity, and there's a lot of interesting possibilities there that are positive. Of course, some things are, uh, are negative, and having a good dialogue about that is great, and I'm, I'm great that people like you are at the center of that dialogue, so it's, it's awesome.

Dawn Song 1:20:54 Right. I think also, um, I also can understand. I think, actually in the past, especially in the past couple of years, um, this rising awareness has been helpful. Like users are also more and more recognizing that privacy is important to them. They should, um, maybe, right, they should be owners of their data. I think this definitely is very helpful. And I think also this type of, uh, voice, also h- a- and together with regulatory framework and so on, also help the companies to essentially put these type of issues at a higher priority. A- and knowing that, uh, right, also it is their responsibility too, to ensure that users are well-protected. Uh, so I think definitely the rising voice is super helpful, and I think that actually really has brought the issue of data privacy, and even this consideration of data ownership to the forefront, to really much wider community. Um, and I think more of this voice is needed, but I think it's just that we want to have a more constructive d- dialogue, um, b- to bring the both sides together, to figure out a constructive solution.

Lex Fridman 1:22:14 So, another interesting space where security is really important is in, in the space of, uh, any kinds of transactions, but it could be also digital currency. So, can you maybe talk, uh, a little bit about blockchain, uh, can you tell me what is a blockchain?

Dawn Song 1:22:30 (laughs) Um, I think the blockchain word itself is actually very overloaded.

Lex Fridman 1:22:37 Of course.

Dawn Song 1:22:37 Uh, i- in general-

Lex Fridman 1:22:39 It's like AI.

Dawn Song 1:22:40 Right. (laughs) Yes. So, in general when we talk of blockchain, we refer to this, uh, distributed ledger in a decentralized fashion. So essentially, you have, um, an, a community of nodes that come together and even though each one may not be trusted, and as long as certain thresholds of the set of nodes, um, uh, behaves properly, then, uh, the system can essentially achieve certain properties. For example, uh, in the distributed ledger setting, you have, you can maintain an immutable log and, uh, you can, uh, ensure that, uh, uh, for example, the transactions actually are agreed upon and then it's immutable and so on.

Lex Fridman 1:23:28 So, first of all, what's a ledger? So, it's a...

Dawn Song 1:23:30 It's like a database, it's like a data entry.

Lex Fridman 1:23:33 And so a distributed ledger is something that's maintained across or is, uh, synchronized across multiple sources, multiple nodes.

Dawn Song 1:23:41 Multiple nodes, yes.

Lex Fridman 1:23:43 And so where is this idea, now how do you keep... So g- so it's important a ledger, a database to keep that, uh, to make sure, so what are the kinds of security vulnerabilities that you're trying to protect against in the context of dist- of a distributed ledger?

Dawn Song 1:24:04 So, in this case, for example, you don't want some malicious nodes to be able to change the transaction logs, uh, and in certain cases called double spending, like, uh, you'll also cause, you can also cause different views in different parts of the network and so on.

Lex Fridman 1:24:22 So, the ledger has to represent, if you're capturing like financial transactions, it has to represent the exact timing and the exact occurrence and no duplicates, all that kind of stuff, it has to be, uh, represent what actually happened. Okay, so what are your thoughts on the security and privacy of digital currency? I can't tell you how many people write to me to interview various people in the digital currency space. There seems to be a lot of excitement there, uh, and it seems to be, some of it's, to me, from a outsider's perspective seems like dark magic. I don't know how secure... I, I think the f- the foundation from my perspective of digital currencies, that is you can't trust anyone, so you have to create a really secure system. So, can you maybe speak about how, what your thoughts in general about digital currency is, and how you, how it can possibly create financial transactions and, uh, financial stores of money in the digital space?

Dawn Song 1:25:31 So, you asked about security and privacy. And so, so again, as I mentioned earlier, in security, we actually talk about two main properties. Um, the, uh, integrity and, uh, confidentiality. Also there's another one for availability. Uh, you want the system to be available.

Lex Fridman 1:25:50 Right.

Dawn Song 1:25:50 But here, uh, for the question you asked, let's just focus on, uh, integrity and confidentiality.

Lex Fridman 1:25:56 Yes.

Dawn Song 1:25:57 So, so for integrity of this distributed ledger essentially as we discussed, we want to ensure that the different nodes, uh, s- s- r-right, so they have this consistent view, usually it's done through, uh, we call a consensus protocol, and that they establish, uh, this shared view on this ledger, uh, that you cannot go back and change-

Lex Fridman 1:26:21 Mm-hmm.

Dawn Song 1:26:21 ... is immutable, uh, and so on. So, uh, so in this case then, the security often refers to, uh, this integrity property, and essentially you're asking the question, how much work, how, uh, how can you attack the system-

Lex Fridman 1:26:38 Mm-hmm.

Dawn Song 1:26:38 ... so that, uh, the attacker can change the log, for example?

Lex Fridman 1:26:46 Right. How hard is it to make an attack like that, yeah?

Dawn Song 1:26:48 Right, right. And then that very much depends on the, the consensus mechanism, the, how the system is built, and all that. So, there are different ways to build these, uh, decentralized systems, and people may have heard about the terms called like proof of work, proof of stake, these different mechanisms, and really depends on, uh, how, right, how the system has been built and also how much resources, how much work has gone into the network to actually, um, say how secure it is. So, for example-

Lex Fridman 1:27:25 Mm.

Dawn Song 1:27:25 ... people talk about like in Bitcoin is proof of work system, so much e- electricity has been burned.

Lex Fridman 1:27:32 So, there's differences, uh, there's differences in the different mechanisms and the implementations of a distributed ledger used for digital currency. Uh, also there's Bitcoin, there's, uh, uh, f- whatever, there's so many of them and there's underlying, uh, different mechanisms, and there's arguments I suppose about which is more effective, which is more secure, uh, which is more, um-

Dawn Song 1:27:52 And what is needed, w- uh, what amount of resource is needed to be able to attack the system. Like for example, what percentage of the nodes do you need to control or compromise in order to, um, right, to change the log, for example.

Lex Fridman 1:28:10 And those are things d, uh, do you have a sense if those are things that could be shown theoretically through the design of the mechanisms? Or does it have to be shown empirically by having a h- uh, a large number of users using the currency?

Dawn Song 1:28:23 I see. So, in general, for each consensus mechanism, you can actually show, uh, theoretically what is needed to be able to attack the system. Uh, of course there are, there can be different types of attacks as we e- um, discuss at the beginning. And so that, um, it's difficult to give, um, like, you know, a complete estimate like really how much is needed to compromise the system. Um, but in general, right, so there are ways to say what percentage of the nodes you need to compromise, uh, and so on.

Lex Fridman 1:29:03 So, we talked about integrity, uh, so on the security side, and then you also mentioned c- the privacy or the confidentiality side. Does it have some of... Uh, does it have some of the same problems and therefore some of the same solutions that you talked about on the machine learning side with differential privacy and so on?

Dawn Song 1:29:23 Mm-hmm. Yeah. So actually, in general on the public, uh, ledger, uh, in this public decentralized systems, um, actually nothing is private. So, all-

Lex Fridman 1:29:35 Right.

Dawn Song 1:29:35 ... the transactions, posts on the ledger, anybody can see. So, in that sense, there's no confidentiality.

Lex Fridman 1:29:42 Mm-hmm.

Dawn Song 1:29:43 Um, so usually what you can do is then, um, there are other mechanisms that you can build in to enable confidentiality, uh, privacy of the transactions and the data and so on. That's also some of the work, um, that, um, both my group and also my startup, uh, does as well.

Lex Fridman 1:30:04 What's the name of the startup?

Dawn Song 1:30:05 Oasis Labs.

Lex Fridman 1:30:06 Oasis Labs. And so the confidentiality aspect there is even though the transactions are public, you want to keep some aspect confidential of the identity of the people involved in the transactions? Or what, what is their hope to keep confidential in this context?

Dawn Song 1:30:25 So, in this case, for example, you want to enable like priv- uh, confidential transactions even, uh, so, so there are different, um, essentially types of data that you want to keep, uh, private or confidential. And you can utilize different technologies including