Anca Dragan: Human-Robot Interaction and Reward Engineering | Lex Fridman Podcast #81

Anca Dragan: Human-Robot Interaction and Reward Engineering | Lex Fridman Podcast #81

Lex Fridman PodcastMar 19, 20201h 38m

Lex Fridman (host), Anca Dragan (guest), Narrator

Anca Dragan’s path into robotics, AI, and human-robot interactionExpressive robot motion and anthropomorphism (e.g., WALL-E, Boston Dynamics)Human modeling: rationality, inverse reinforcement learning, and intuitive physicsHuman-robot collaboration as a game-theoretic and underactuated control problemAutonomous driving: interacting with human drivers, pedestrians, and semi-autonomous systemsReward design and reward learning (mis-specification, Goodhart’s law, leaked information)Ethical, social, and philosophical questions: how we treat robots, mortality, and meaning

In this episode of Lex Fridman Podcast, featuring Lex Fridman and Anca Dragan, Anca Dragan: Human-Robot Interaction and Reward Engineering | Lex Fridman Podcast #81 explores designing Robots That Understand Human Intent, Limits, and Preferences Lex Fridman and Anca Dragan discuss human-robot interaction with a focus on how robots can model, predict, and adapt to human behavior and preferences. They explore inverse reinforcement learning, rationality assumptions, and how to reinterpret human 'irrationality' as optimal behavior under different beliefs, constraints, or physics models. A major theme is reward design: how hard it is to specify objectives that elicit the right behavior in all situations, and how robots can learn from 'leaked' information in human actions, corrections, environment, and even emergency stops. They also touch on autonomous driving, semi-autonomous systems, ethical and social dimensions of robots, and broader reflections on meaning, mortality, and what it means to build AI that truly serves humans.

Designing Robots That Understand Human Intent, Limits, and Preferences

Lex Fridman and Anca Dragan discuss human-robot interaction with a focus on how robots can model, predict, and adapt to human behavior and preferences. They explore inverse reinforcement learning, rationality assumptions, and how to reinterpret human 'irrationality' as optimal behavior under different beliefs, constraints, or physics models. A major theme is reward design: how hard it is to specify objectives that elicit the right behavior in all situations, and how robots can learn from 'leaked' information in human actions, corrections, environment, and even emergency stops. They also touch on autonomous driving, semi-autonomous systems, ethical and social dimensions of robots, and broader reflections on meaning, mortality, and what it means to build AI that truly serves humans.

Key Takeaways

Model humans as approximately rational—but under their own beliefs and constraints.

Rather than dismissing people as irrational, robots can treat human behavior as roughly optimal given different world models, planning horizons, or intuitive physics; this shift makes behavior more predictable and supports better assistance and coordination.

Get the full analysis with uListen AI

Use inverse reinforcement learning to infer what people want from what they do.

By assuming actions are (noisily) optimal for some underlying reward, robots can infer user preferences or driving styles from demonstrations and then optimize accordingly, instead of relying solely on hand-specified objectives.

Get the full analysis with uListen AI

Let robots act to gather information, not just passively predict humans.

Robots can nudge, probe, or test the environment (e. ...

Get the full analysis with uListen AI

Treat human behavior as part of an underactuated system you influence but don’t control.

Humans are like degrees of freedom you cannot command directly but can shape through your actions; planning should account for the fact that people change their behavior in response to what robots do, not just vice versa.

Get the full analysis with uListen AI

Reward design is brittle; assume specified rewards are evidence, not ground truth.

Engineers rarely write perfect reward functions—agents can optimize them in unintended ways (Goodhart’s law); robots should treat designer-specified objectives as noisy signals about the true human desiderata and keep uncertainty over what they should optimize.

Get the full analysis with uListen AI

Exploit “leaked” information from humans and the environment to refine rewards.

Physical corrections, emergency stops, demonstrations, verbal feedback, and even the state of an environment (e. ...

Get the full analysis with uListen AI

Semi-autonomous systems must be designed to keep humans meaningfully engaged, not just on-call.

Assuming a supervising driver will perform as well as an active driver is risky; systems should be engineered so the human plus automation combination is safer and more capable than either alone, with explicit attention to human attention and off-distribution states.

Get the full analysis with uListen AI

Notable Quotes

Maybe people are operating this thing, but assuming a much more simplified physics model… and under those assumptions, their behavior actually makes sense.

Anca Dragan

When the robot moves in an optimal way and I intervene, that means I disagree with its notion of optimality.

Anca Dragan

We’ve moved the tuning from the behavior side into the reward side, and it still seems really hard to anticipate every possible situation.

Anca Dragan

Our world is something that we’ve been acting in according to our preferences; the environment itself leaks information about what people want.

Anca Dragan

It’s such a great privilege to exist that the idea of being told I’m going to die is my biggest nightmare.

Anca Dragan

Questions Answered in This Episode

How can we systematically identify when our rationality-based models of humans are breaking down, and what should robots do in those edge cases?

Lex Fridman and Anca Dragan discuss human-robot interaction with a focus on how robots can model, predict, and adapt to human behavior and preferences. ...

Get the full analysis with uListen AI

What are practical strategies for deploying robots that actively probe humans for information without feeling intrusive, manipulative, or unsafe?

Get the full analysis with uListen AI

How should regulators and designers think about assigning responsibility when semi-autonomous systems rely on human supervision in off-policy states?

Get the full analysis with uListen AI

What forms of expressivity or vulnerability in robots are most effective at eliciting respectful, prosocial behavior from humans without deceiving them?

Get the full analysis with uListen AI

If reward functions are always imperfect, what governance or oversight mechanisms should exist around how powerful AI systems learn and update their true objectives over time?

Get the full analysis with uListen AI

Transcript Preview

Lex Fridman

The following is a conversation with Anca Dragan, a professor at Berkeley working on human-robot interaction, algorithms that look beyond the robot's function and isolation and generate robot behavior that accounts for interaction and coordination with human beings. She also consults at Waymo, the autonomous vehicle company. But in this conversation, she's 100% wearing her Berkeley hat. She's one of the most brilliant and fun roboticists in the world to talk with. I had a tough and crazy day leading up to this conversation, so I was a bit tired, even more so than usual. (laughs) But, uh, almost immediately as she walked in, her energy, passion, and excitement for human-robot interaction was contagious. So I had a lot of fun and really enjoyed this conversation. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, review it with five stars on Apple podcast, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. As usual, I'll do one or two minutes of ads now and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Since Cash App does fractional share trading, let me mention that the order execution algorithm that works behind the scenes to create the abstraction of fractional orders is an algorithmic marvel. So big props to the Cash App engineers for solving a hard problem that, in the end, provides an easy interface that takes a step up to the next layer of abstraction over the stock market, making trading more accessible for new investors and diversification much easier. So again, you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10 and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world. And now, here's my conversation with Anca Dragan. When did you first fall in love with robotics?

Anca Dragan

I think it was a very gradual process and it was somewhat accidental actually, because I first started getting into programming when I was a kid, and then into math, and then into comp- I decided computer science was the thing I was gonna do, and then in college I got into AI, and then I applied to the Robotics Institute at Carnegie Mellon. And I was coming from this little school in Germany that no one e- nobody had heard of, but I had spent an exchange semester at Carnegie Mellon, so I had letters from Carnegie Mellon. So that was the only pl- you know, MIT said no, Berkeley said no, Stanford said no. That was the only place I got into, so I went there to the Robotics Institute. And I thought that robotics is a really cool way to actually apply the stuff that I knew and loved, like optimization. So that's how I got into robotics. I have a better story how I got into cars, which is I, you know, I used to do mostly manipulation in my PhD, but now I do kind of a bit of everything application-wise, including cars. And I got into cars because I was here in Berkeley while I was a PhD student still for RSS 2014.

Install uListen to search the full transcript and get AI-powered insights

Get Full Transcript

Get more from every podcast

AI summaries, searchable transcripts, and fact-checking. Free forever.

Add to Chrome