Dawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95
Lex Fridman (host), Dawn Song (guest), Narrator
In this episode of Lex Fridman Podcast, featuring Lex Fridman and Dawn Song, Dawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95 explores dawn Song on hacking AI: vulnerabilities, defenses, and data ownership Lex Fridman and Dawn Song explore computer security across classic software bugs, human-focused social engineering, and emerging attacks on machine learning systems.
Dawn Song on hacking AI: vulnerabilities, defenses, and data ownership
Lex Fridman and Dawn Song explore computer security across classic software bugs, human-focused social engineering, and emerging attacks on machine learning systems.
They discuss adversarial machine learning in depth: how models can be fooled at inference and training time, including physical-world attacks on stop signs, backdoored facial recognition, and black-box attacks on real services like Google Translate.
Song explains parallel privacy risks, showing how trained models can leak sensitive training data and how techniques like differential privacy and confidential computation can mitigate this.
The conversation broadens to data ownership, blockchain-based responsible data economies, program synthesis as a path toward intelligent machines, and philosophical reflections on meaning, creativity, and scientific collaboration.
Key Takeaways
Security vulnerabilities are unavoidable, but their impact can be reduced.
Formal verification and program analysis can prove specific properties (like memory safety) for real systems such as kernels and crypto libraries, yet the vast and evolving space of attack types means no complex real-world system can be guaranteed 100% secure.
Get the full analysis with uListen AI
The security weak point is increasingly human, not just code.
As systems harden, attackers shift “up the stack” to exploit people through phishing, social engineering, fake news, and deepfakes; AI-powered chatbots could act as user-side guardians that monitor conversations, challenge suspicious claims, and even interrogate attackers.
Get the full analysis with uListen AI
Machine learning models can be systematically fooled and backdoored.
Adversarial examples with tiny input perturbations can force misclassification, including robust physical attacks (e. ...
Get the full analysis with uListen AI
Defenses work best when they exploit natural structure and redundancy.
Checks like spatial consistency in images (overlapping patches should yield similar segmentations) and temporal consistency in audio/video make life hard for attackers, and combining multiple sensors or modalities (vision, LIDAR, radar) further raises the bar for successful attacks.
Get the full analysis with uListen AI
Trained models can leak private training data unless designed otherwise.
Even without model internals, attackers can query language models trained on sensitive emails and recover actual Social Security or credit card numbers; training with differential privacy adds controlled noise in learning so models retain utility while sharply reducing such leakage.
Get the full analysis with uListen AI
Clarifying data ownership is key to a healthy digital economy.
Song argues that, as with physical property rights historically driving economic growth, we need clear ownership and enforceable policies for personal data so individuals can decide how their data is used, traded, or monetized instead of platforms implicitly owning everything.
Get the full analysis with uListen AI
Program synthesis is a promising testbed for advancing AI toward generality.
Teaching machines to write code—from simple IFTTT-style rules to SQL and recursive programs—both enables practical tools (e. ...
Get the full analysis with uListen AI
Notable Quotes
“Security is job security.”
— Dawn Song
“We are still at a very early stage of really developing robust and generalizable machine learning methods.”
— Dawn Song
“It’s almost impossible to say that a real world system is 100% no security vulnerabilities.”
— Dawn Song
“The weakest link of the system is oftentimes humans themselves.”
— Dawn Song
“Once we teach computers to write software—to write programs—then I guess computers will be eating the world by transitivity.”
— Dawn Song
Questions Answered in This Episode
How can we realistically deploy AI “security agents” that protect users across all their online interactions without creating new privacy risks?
Lex Fridman and Dawn Song explore computer security across classic software bugs, human-focused social engineering, and emerging attacks on machine learning systems.
Get the full analysis with uListen AI
What would a practical, user-friendly data ownership system look like, where individuals truly control and monetize their data day to day?
They discuss adversarial machine learning in depth: how models can be fooled at inference and training time, including physical-world attacks on stop signs, backdoored facial recognition, and black-box attacks on real services like Google Translate.
Get the full analysis with uListen AI
How should regulators and industry balance openness in AI research with the potential for real-world adversarial attacks on deployed systems?
Song explains parallel privacy risks, showing how trained models can leak sensitive training data and how techniques like differential privacy and confidential computation can mitigate this.
Get the full analysis with uListen AI
What benchmarks or breakthroughs in program synthesis would convince you that we’re genuinely closer to artificial general intelligence rather than narrow tools?
The conversation broadens to data ownership, blockchain-based responsible data economies, program synthesis as a path toward intelligent machines, and philosophical reflections on meaning, creativity, and scientific collaboration.
Get the full analysis with uListen AI
In safety-critical domains like autonomous driving, what minimum robustness and multimodal consistency standards should be required before large-scale deployment?
Get the full analysis with uListen AI
Transcript Preview
The following is a conversation with Dawn Song, a professor of computer science at UC Berkeley, with research interests in computer security, most recently, with a focus on the intersection between security and machine learning. This conversation was recorded before the outbreak of the pandemic. For everyone feeling the medical, psychological, and financial burden of this crisis, I'm sending love your way. Stay strong. We're in this together. We'll beat this thing. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. As usual, I'll do a few minutes of ads now, and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Since Cash App does fractional share trading, let me mention that the order execution algorithm that works behind the scenes to create the abstraction of fractional orders is an algorithmic marvel. So big props to the Cash App engineers for solving a hard problem that, in the end, provides an easy interface that takes a step up to the next layer of abstraction over the stock market, making trading more accessible for new investors and diversification much easier. So again, if you get Cash App from the App Store or Google Play, and use the code LEXPODCAST, you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world. And now, here's my conversation with Dawn Song. Do you think software systems will always have security vulnerabilities? Let's start at the broad, almost philosophical level.
That's a very good question. I mean, in general, right, it's very difficult to write completely bug-free code, uh, and code that has no vulnerability, and also especially given that the definition of vulnerability is actually really broad. It's any type of attacks, uh, essentially on the code can, you know, that's- can, you can call that, uh, the cause by vulnerabilities.
And the nature of attacks is always changing as well?
Right.
Like new ones are coming up?
Right. So for example, in the past, we talked about memory safety type of vulnerabilities, where, uh, essentially attackers can exploit, um, the software and then take over control of how the code runs, and then can launch attacks that way.
By accessing some aspect of the memory, and be able to then, uh, alter the state of the program?
Exactly. So for example, in the example of a buffer overflow, then the- the attacker essentially actually causes, uh, essentially unintended changes in the state of the- of the program, and then, for example, can then take over control flow of the program and lead the program to execute, uh, codes that actually they- the programmer didn't intend. So the attack can be a remote attack. So the- the attacker, for example, can- can send in a malicious input to the program that just causes the program to completely then be compromised and then end up doing something that's under the program- uh, under the attacker's control and, uh, intention. But that's just one form of attacks, and there are other forms of attacks. Like, uh, for example, there are these side channels where attackers can try to learn from, uh, even just observing the outputs, uh, from the behaviors of the program, try to infer certain secrets of the program. So they, uh, essentially, right, the form of attacks is very varied. It's very broad, uh, spectrum. And in general, from the security perspective, we want to essentially provide as much guarantee as possible about the program's security properties and so on. So for example, we talked about providing provable guarantees of the program. Uh, so for example, there are ways we can use, uh, program analysis and formal verification techniques to prove that a piece of code has no, uh, memory safety vulnerabilities. Um-
Install uListen to search the full transcript and get AI-powered insights
Get Full TranscriptGet more from every podcast
AI summaries, searchable transcripts, and fact-checking. Free forever.
Add to Chrome