Interpretability and Safety for Robot Foundation Models

1. 0:00 – 0:07

   Intro

   1. AGAnkit Gupta0:00

      [upbeat music]

2. 0:07 – 0:30

   Why robot foundation models need interpretability now

   1. AGAnkit Gupta0:07

      We're here at YCML with Bear, who's gonna talk about his work from Berkeley on robot AI safety. He's also the founder of the Physical AI Safety Institute. Bear, tell us about your work.

   2. BHBear Häon0:18

      Awesome. Uh, so this is a piece from, uh, Carl last year, uh, where we wrote the first ever interpretability paper for Robot Foundation Models. Um, and, uh, the-

   3. AGAnkit Gupta0:28

      It's crazy that the first one was last year.

3. 0:30 – 1:10

   From “AI safety for digital agents” to safety for embodied robots

   1. BHBear Häon0:30

      It's, it's insane. Uh, there's been about 30 pieces of work since that have built upon this, and they've done what, uh, I love to say is, like, a much more sophisticated approach, from SAEs to looking at world action models, et cetera. Um, but this very first pass was to really get this, uh, just this more concrete resolution of could you take techniques from the AI safety field, which is, for be- about a decade, been building techniques across interpretability, alignment, and control, uh, for essentially systems that just think and, and maybe digitally act. But could we take those techniques and apply them to create concrete safety outcomes for systems that both think and physically act?

   2. AGAnkit Gupta1:08

      Cool.

   3. BHBear Häon1:08

      Um-

   4. AGAnkit Gupta1:08

      Yeah, so what's the, what's the essence of how you did that? Because-

4. 1:10 – 1:19

   What the model is: VLA as a fine-tuned vision-language model for action

   1. BHBear Häon1:10

      Yeah, so the essence of how we did this is, uh, we start with a VLA, and a VLA is, for all intents and purposes, uh, uh, a fine-tuned, uh, VLM.

   2. AGAnkit Gupta1:19

      Yep.

5. 1:19 – 1:50

   Peering inside transformers: FFNs as concept “dictionaries” via token activations

   1. BHBear Häon1:19

      Um, and within the VLA are transformer blocks. Uh, and within transformer blocks are feedforward networks, which are essentially just an MLP. So if we take, uh, if we project that onto the model embedding layer, which is the, the, the final layer of any, you know, most foundation models, it's essentially the, the dictionary of the tokens the model understands, um, you actually get this representation that's almost like a dictionary of, for every FFN, what were the tokens across all the FFNs that were most activated by that FFN, uh, layer? Now,

6. 1:50 – 2:05

   Clustering internal features into concepts (fast/slow/cautious)

   1. BHBear Häon1:50

      this allows you to do a lot of really interesting analysis to understand RFMs better, but what's really useful here is you can then cluster these FFN representations by similar concepts, whether that's quick, fast, slow, cautious, et cetera,

7. 2:05 – 2:21

   Steering robot behavior by hyper-activating neuron clusters during inference

   1. BHBear Häon2:06

      and then in the forward pass of the RFM, actually hyper-activate that so that the model, uh, is, you know, very, uh, concretely controlled toward that direction. Um, and so I've got a copy of our, uh, website here, uh,

8. 2:21 – 2:35

   Demo outcome: same command, different action-space behavior (high vs low activation)

   1. BHBear Häon2:21

      where you can see here we have a low neuron cluster activated and a high neuron cluster activated.

   2. AGAnkit Gupta2:26

      Interesting.

   3. BHBear Häon2:26

      And it has the same natural language command. However, the robot is achieving the task in a much higher or lower action space, um, because of this very direct control.

9. 2:35 – 3:28

   How interpretability connects to safety: constraining unexpected harmful behavior

   1. AGAnkit Gupta2:35

      And so when I think about this from the lens of AI safety-

   2. BHBear Häon2:37

      Yeah

   3. AGAnkit Gupta2:38

      ... is the idea here that this means, you know, if we can understand these elements inside the neural networks better, we can constrain them in such a way such that when you have a generic command, like the command you're doing here, it's less likely to do something unexpectedly bad. Is that, like, roughly why you see this as, like, a AI safety problem? Or, like, how do you interpret it that way?

   4. BHBear Häon2:59

      Yeah. You know, I think people approach AI safety from a whole, you know, sort of, uh, fidelity of different normative understandings. Anywhere from, "We just want our systems to be more reliable," to, "There could be serious catastrophic risks here." And, uh, uh, my opinion and the opinion of the, you know, the authors of the paper and the, the Physical AI Safety Institute is that when you, uh, are, when you have a general reasoning model that is operating on probability distributions of what anything really semantically means in the real world-

   5. AGAnkit Gupta3:28

      Yeah

10. 3:28 – 4:31

    Why embodiment escalates stakes: from knowledge to physical capability

    1. BHBear Häon3:28

       ... it, it's an extra surface of risk that i, that sort of, uh, uh, o- only amplifies, uh, those existing risks that the AI safety community thinks of. Um, so whether, you know, whether it is simply just, will this do the task that I want it to do? Whether it is, uh, you know, a long horizon task that might go very off-stream because, uh, the model has, you know, really improved over time but now you're not so sure if it has, you know, the inner alignment tracing, um, or, um, if it's, you know, some catastrophic maybe a bio risk of, uh, of an RFM. Now, you know, you could say an LM might, uh, know how to make, uh, a bioweapon, um, but an RFM embedded in a humanoid can, can, uh, know how to make it. It can go buy the parts by walking into Home Depot in a world that's, you know, very normalized that to have, you know, robot embodiments, and then actually go make it and place it in a high-density location. And so there's a whole, uh, sort of remit of, of risks in the space. Um, and our goal really with, uh, the Physical AI Safety Institute, um, is to get more people working on technical solutions to these problems.

11. 4:31 – 5:36

    Building a field: the ‘Science of Physical AI Safety’ workshop and its agenda

    1. AGAnkit Gupta4:31

       Nice. Um, maybe to wrap, you had mentioned that you're hosting a workshop-

    2. BHBear Häon4:35

       Yeah

    3. AGAnkit Gupta4:35

       ... uh, next year. Do you wanna tell us a little about that?

    4. BHBear Häon4:37

       Yeah. So the workshop is this year in November.

    5. AGAnkit Gupta4:39

       Sorry, this year.

    6. BHBear Häon4:40

       Um, from August 12th, you can, um, you can, uh, submit, uh, a, a paper. Um, so the workshop is called The Science of Physical AI Safety. The goal is to bring together the AI safety and robot learning communities to establish this shared field. We have three of the most fantastic robotics professors in the world from Stanford, CMU, and MIT, as well as Thomas from Goodfire, uh, who's done fantastic work on, on physical AI, uh, interpretability as well. Um, and really, uh, the idea with the workshop is to, as a group, collectively source opinions to these three questions, and the three questions are: What can we take from the AI safety field to inform RFMs? What can we take from the classical robotics fields of kinematics, dynamics, and control to inform safety for RFMs? Um, and also, how should we think about evaluating RFMs in contrast to these two disciplines? Um, it opens August 12th, so we'd love for people to, to submit, um, either a paper, um, or a short demo.

    7. AGAnkit Gupta5:32

       Awesome. Thanks so much.

    8. BHBear Häon5:34

       Cool. Appreciate your time, man.

    9. AGAnkit Gupta5:34

       Really appreciate the time. Thanks. Appreciate it.

Episode duration: 5:36