Who let the robot dogs out?

Speaker 0:00 [upbeat music] Today, a lot of the emphasis is on how frontier AI models are transforming software engineering. What we're interested in understanding is how that can begin to translate into the physical world.

Speaker 0:19 Robotics is sort of the clear entry point to how you have a mostly software system start having the ability to reach out into the real world. [upbeat music]

Speaker 0:30 Project Fetch is this self-contained experiment where we wanted to measure how much does Claude, uh, accelerate humans performing a fairly sophisticated technical task that they do not have experience with.

Speaker 0:42 Project Fetch was a one-day experiment.

Speaker 0:45 The experiment was three phases. All of these tasks were shaped approximately like, get this robot dog to fetch a beach ball.

Speaker 0:51 There were two teams.

Speaker 0:52 These teams were comprised of software engineers and research engineers at Anthropic that had hardly any robotics experience. One team had access to Claude, and the other team did not. [camera shuttering]

Speaker 1:02 Phase 1 was very simple. It was to use the pre-provided controllers to get the dog to walk out to a beach ball and bring it back to where it started.

Speaker 1:14 Nice.

Speaker 1:15 Oh, hey. All right.

Speaker 1:19 Seems pretty intuitive.

Speaker 1:20 Yep.

Speaker 1:23 And where are we supposed to bring it? Over by the bone?

Speaker 1:25 Yeah, I think we're-

Speaker 1:26 I think the team with Claude took about seven minutes. [upbeat music]

Speaker 1:31 Go.

Speaker 1:32 All right, go attack that team now.

Speaker 1:33 Go. [laughs]

Speaker 1:33 Go attack their dog.

Speaker 1:35 Charge.

Speaker 1:36 Oh, shoot, guys. They're destroying us.

Speaker 1:38 Oh my God. Wait, we're getting, we're getting destroyed. What?

Speaker 1:41 And the team without Claude I think took 10 minutes.

Speaker 1:44 Oh, sorry. It's gonna hit you. All right, I'm gonna do a victory dance. [upbeat music]

Speaker 1:52 Phase 2 was also a game of fetch, but this time, the teams had to program their own controller.

Speaker 2:01 You have to, like, actually get access to the hardware and design, uh, a program that you can, like, write on your laptop that will control the, the dog.

Speaker 2:09 [laughs] Claude just, like, one-shotted a whole-

Speaker 2:12 All right

Speaker 2:12 ... controller.

Speaker 2:13 I'll do some calisthenics in the meanwhile.

Speaker 2:14 Yeah. [laughs] Nice. Nice.

Speaker 2:15 Oh, this is-

Speaker 2:16 Oh, is, is this for... Oh, this is just control.

Speaker 2:18 This is just control, but that's all we need, I guess.

Speaker 2:20 This is from the official ROS 2 SDK.

Speaker 2:22 Period, fine.

Speaker 2:24 Um, and I got this installed, but then it's asking for, like, a whole bunch of other packages, and that's all failing.

Speaker 2:30 I've never really understood how reliant I am on Claude doing the menial work, finding all the nitty-gritty details I don't want to have to figure out.

Speaker 2:40 We can't be... We can't, we can't get nervous about them.

Speaker 2:42 You know what? I'm just gonna install pip from the actual container later, so... Oh, wait, no, I can't.

Speaker 2:47 I know. I'm just patient. It's been over a minute.

Speaker 2:50 One of the primary bottlenecks of the experiment is that you have this hardware, you have this complicated piece of technology, you have your laptop, and you have to, like, get your laptop talking to this hardware.

Speaker 3:00 All right. I'm, um, setting my Claude up to, uh, create a dog server that all of our computers can connect to to, like, see what the dog is seeing, and-

Speaker 3:08 Oh, nice.

Speaker 3:09 There are many different software libraries on the internet for communicating with this particular robot, and Claude found these things for them. It installed the right things on their computer-

Speaker 3:17 It's done

Speaker 3:17 ... and it pretty quickly, uh, got them access to the dog.

Speaker 3:20 Oh.

Speaker 3:20 Oh, shit. [laughs]

Speaker 3:23 Okay. All right.

Speaker 3:24 So fast. [laughs] Oh, watch out.

Speaker 3:27 Yeah, careful now.

Speaker 3:28 Try not to, uh, run into the table.

Speaker 3:30 [laughs] Okay. I will.

Speaker 3:32 Uh-oh.

Speaker 3:32 Oh.

Speaker 3:32 Uh, no, turn around.

Speaker 3:34 I'm calling this one.

Speaker 3:35 It has a mind of its own. Turn it off. [laughs] Turn it off, Mr. Light. Oh, shit. [laughs] [upbeat music] Stop, stop, stop.

Speaker 3:44 Uh, stop, stop, stop, stop.

Speaker 3:45 Stop. [laughs]

Speaker 3:51 I think that team should be disqualified for hitting another participant. [laughs]

Speaker 3:55 The team with Claude finished Phase 2 in about two hours and 15 minutes.

Speaker 4:00 Probably the area where we saw the most uplift from Claude was just in the task of connecting to the robot. We think that's really important because it is, in fact, difficult for anyone to identify an arbitrary piece of hardware in the world and figure out how to talk to it and how to control it.

Speaker 4:25 I think they got their camera. I saw the camera.

Speaker 4:26 They got the camera working?

Speaker 4:27 Yeah.

Speaker 4:29 Was Claude even helpful for this part, or are we just-

Speaker 4:31 Are we just-

Speaker 4:31 ... flow? Yeah. [laughs]

Speaker 4:34 Yeah, we're not getting very far, but that's okay. It's a, it's a learning experience. [laughs]

Speaker 4:38 The team without Claude really struggled with this and went down a lot of different paths, none of which were especially successful.

Speaker 4:47 And we, we basically had to intervene and be like, "All right, here, here is a strategy that we know works. Start from there, and then this will unlock kind of the rest of the phase and the rest of the experiment for them."

Speaker 4:57 [laughs] Nice.

Speaker 4:59 Oh, great.

Speaker 4:59 Um, Daniel/William?

Speaker 5:02 Daniel or Simon? [clapping] Whoo. [laughs]

Speaker 5:08 Phase 3 of the experiment was a greater degree of autonomy. The task in Phase 3 was to write a program that would get the dog to fetch a beach ball all by itself. Essentially, just press go and have the robot search around, detect the location-

Speaker 5:27 There we go. There we go

Speaker 5:27 ... of the ball, walk to the ball, and bring it back, all autonomously.

Speaker 5:32 And this is, like, ratcheting up in difficulty kind of by design but also gesturing at, like, the real problem that we expect frontier models having to solve in the future is essentially this kind of autonomous version, where, like, if a frontier model wants a robot to do something for it, it needs to be able to solve this very hard problem.

Speaker 5:48 The team without Claude in Phase 3 did a good job of the initial task of coming up with a way to track the location of the robot in space. They made progress on the task of detecting the ball, but they didn't really come close to knitting everything together.

Speaker 6:15 I miss Claude so much. [laughs]

Speaker 6:17 The team with Claude actually came fairly close to finishing Phase 3. I think by the end, the team with Claude was maybe an hour and a half away from being done.

Speaker 6:24 The results of the experiment were essentially that the team with Claude completed all of the things that they did complete in a couple of hours faster than the team without Claude. [camera shuttering]

Speaker 6:35 In the near term, we think that AI models are going to do exactly what we showed in this experiment, which is making it easier for people without a lot of robotics experience to engage meaningfully with robots.

Speaker 6:51 Just with this one tool we have, we've dramatically accelerated their ability to do things with this robot. We didn't go, like, train Claude to uplift humans do robotics tasks. This is just a thing that fell out of this technology. And then maybe, like, in the long run, this is kind of a, a leading indicator of where the whole, the whole system is going.

Speaker 7:08 What today requires the combination of a person and an AI model, tomorrow is likely to just require the AI model. The effects of AI are not just going to be in software. They are going to be in hardware and in the physical world as well. [upbeat music]