By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceRobots That Predict the Future: Dyna-2 & Physical AI — Jason Ma, Dyna Robotics
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
EVERY SPOKEN WORD
80 min read · 15,840 words- 0:00 – 1:56
Hook
- JMJason Ma
Yeah. So if we wanna learn a new task, uh, you know, we basically do a little bit of robot data collection. Then the model, these-
- SWswyx
What is a little bit like?
- JMJason Ma
Yeah, like couple hours.
- SWswyx
Couple hours.
- JMJason Ma
Yeah. We're, in some of the tasks, like, just tens of minutes, right? Yeah. And then-
- SWswyx
The chopping one was 45 minutes.
- JMJason Ma
Yeah.
- SWswyx
The open the fridge was on- And that's an interview, that's like an onboarding task for one of your engineers.
- JMJason Ma
Our robots today fold thousands of napkins for, uh, many, many different locations, many different Din Tai Fung locations in the United States. So that's actually our biggest scaled customer, right? So I wanna make a distinction here. Um, today there's a lot of robotics companies just talking about deployments, talking about pilots, but what's really hard is not putting a robot in one location or doing a pilot. What's really hard is that the customers believe or validate that your robot creates enough value that they wanna deploy your robots across many of their sites. So that's what's happened for us with Din Tai Fung, which to the best of my knowledge, hasn't really happened for other companies doing similar, like, uh, general purpose robots. So at Din Tai Fung, our robots fold thousands of napkins per day.
- SWswyx
Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately enough of you actually subscribe to us to keep all this sustainable without ads, and we wanna keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring Latent Space to you each and every week. If you do it, I promise you we'll never stop working to make the show even better. Now let's get into it.
- 1:56 – 5:04
Introduction and Dyna-2's World-Action Model
- SWswyx
[upbeat music] All right, we're here on site with Jason Ma, the, uh, co-founder, chief scientist of Dyna Robotics. Thanks for having us in your-
- JMJason Ma
Of course. Thank you for coming.
- SWswyx
Yeah. It's been a blast. We got to see-- You'll see a lot of B-roll, but we got to see, uh, drinks being pulled, a lot of evals, robots with knives for the first time.
- JMJason Ma
[laughs]
- SWswyx
Uh, so under the hood of all this, you have a world-action model. What is that? [laughs]
- JMJason Ma
Yeah. So last month we released our world-action model called Dyna-2. So it's the first world-action model trained on more than one million hours of data. So going back to your question first, world-action model is a model that's able to predict the future video, the future state of the world, so hence the name world model. But in addition, you can also use that future generation to also produce actions robots can execute in the physical world to control the robot. So why do we do that? It's a very powerful architecture because if you think about how humans act in the physical world, in our brain we're implicitly thinking about how the world will evolve based on our actions. So now we have a model architecture that's able to do that to guide robot actions. So what we find is that by doing so, the model is much more dexterous, much more robust, and much more capable of generalizing to environments, to scenarios it's never seen before.
- SWswyx
Yeah.
- JMJason Ma
Yeah.
- SWswyx
Uh, I wanna prompt this question a bit. I think a lot of people hear world model, and they think you're outputting video or world or 3D state. Um, you're also doing action. I think most people don't understand what models are under the hood of robotics. So what is input, output? What's the training data? What is the architecture?
- JMJason Ma
Yeah.
- SWswyx
Give us all the details. [laughs]
- JMJason Ma
Yeah. Yeah. So, uh, for inputs, the most important input is video, right? You know, you can think about videos that a robot is capturing with its cameras, right? So video or images of the, you know, past many seconds, that's the input. But for robots, what's really interesting is that robot also has a lot of actuator, a lot of, uh, joint encoders on itself. So the robot also knows, you know, the angles of its joints. So those can also be inputs to this world model. That's something that's very different from just a pure video model that's only looking at videos, right? So that's the input. And additionally, for robots, you want to be able to talk to it, to instruct it to do different tasks, right? So you can also feed it like language instruction. So those are what I consider the basic set of inputs that go into any robotics models, including world-action models. And once we have that, the output at the, uh, basic level, it needs to output actions the robots can execute on the hardware. But in the case of world-action model, what's really interesting is that in addition to the actions, you can also output future video, right? So basically, you can almost output the future video or some representation of it first, and then use that to guide robot actions. So the way I think about it is like, this is kinda analogous to, you know, chain of thought, you know. Basically, the robots, the models are reasoning in the visual, the physics space, and then outputting actions. So that's the key difference between world-action
- 5:04 – 10:15
Diffusion Transformers, Flow Matching, and Generalist Robots
- JMJason Ma
models outputs versus other kind of models, robot models outputs.
- SWswyx
Um, you put out a good blog post around your model. I think a lot of people understand language models or transformers, next token predictors. In this case, you shared quite a bit. There's like diffusion, flow matching. You're predicting next world state plus action. Um, one thing you say is a million hours. How do you determine the architecture? Like what are the choices there? What is the architecture? I feel like in language models there's a lot of take something off the shelf-
- JMJason Ma
Yeah
- SWswyx
... take trillions of tokens, do prediction, SFT, RL, and you have a model. What about this? How do you go from nothing to your model? Is there vanilla architecture off the shelf? Do you determine all this? And what's, what is the current state?
- JMJason Ma
Yeah, I think, yeah, so the mental framework to think about how do we build models for robotics is that there are essentially a couple things to look at. One is data, and the second is like- How do we make a model learn as much as possible from that data? So let's first talk about the model actually, right? So the blog post we released is our Dyna-2 world-action model. So under the hood, it's a diffusion transformer. So transformer is essentially like the language model, you know, uh, uh, sequence to sequence match-
- AFAlessio Fanelli
Decoder only
- JMJason Ma
... right. Yeah. But what's really interesting is that in visual domains, right? If you think about generating images, generating videos, and in this case, robots looking at images to generate actions, people have found the diffusion kind of architecture to work really well. So what is a diffusion model? A diffusion model is a model that takes some random noise and just through iterative computation can go from noise to something that's really high dimensional, that's something really highly, like, defined and have very fine grain feature. If you think about generating images, right? You want sharp edges, sharp corners, color contrast. And for robotics, even though you're not generating image, you're generating actions that needs to be very precise, very, like smooth, right? So it has kind of the same kind of, uh, consideration or desiderata as generating image. So that's why from first principle, you know, the whole field, not just us, have been using this kind of diffusion transformer architecture to train robotics foundation models. You need the transformer aspect because transformer is the thing that can just ingest the most amount of data, can learn. Then diffusion helps you to refine the outcome, to get to the smooth, precise, fine grain motion you need to actually execute on the robot. So that's the architecture that goes behind a lot of our recent breakthroughs.
- AFAlessio Fanelli
And we, and we've covered flow matching before on the podcast. It is mostly an efficiency speed up. Like let's say you normally would take 40 to 100 diffusion steps, you can reduce it to four.
- JMJason Ma
Yeah, that's right. So with a, uh, diffusion model, one thing that's really hard in practice is that you need to tune the scheduler. A lot like by how much you're tuning or converting the initial noise input into the final output. So flow matching is kind of like a special case of diffusion that makes the transformation much simpler. In practice, you don't have to tune as much, and it works just as well. That is why I think about two to three years ago, most generative modeling for image and video has shifted from a diffusion model to a flow matching model. But the concept is the same, and flow matching is just a special, simpler to tune, uh, variant of diffusion. Yeah.
- AFAlessio Fanelli
It actually makes me think that, so flow matching obviously has trade-offs in quality-
- JMJason Ma
Yeah
- AFAlessio Fanelli
... but actually it's very suited for action diffusion and, and, and robotics because you get to reevaluate in real time.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Whereas in, in images, you're... it's basically one shot.
- JMJason Ma
That's right.
- SWswyx
Which is pretty fun. We got to mess with the laundry folding robot. Like, you know-
- JMJason Ma
Yeah
- SWswyx
... throw a random towel in-
- JMJason Ma
Yeah
- SWswyx
... um, the models adapt, right?
- JMJason Ma
Yeah.
- SWswyx
I think one key thing about your models is that they're tr- they're made to be generalists, so one foundation model and you adapt them a lot.
- JMJason Ma
Yeah.
- SWswyx
Uh, how does that work? Basically, in 45 minutes of mapped data, you can generalize towards... We saw an example of, um, pulling drinks out of a fridge.
- JMJason Ma
Yeah.
- SWswyx
What's the, what's the mindset there of one general model, and then how do you adapt to different tasks?
- JMJason Ma
Yeah. So the way we think about it is that, uh, for robotics, we need to build a very general strong base model, right? And then because if you think about how a human interacts and behaves in the physical world, right? We have very strong base knowledge. We can just readily do a lot of tasks. But if you want to become an expert at any, any given task, there is some amount of training that needs to happen. And I think that training general purpose robots, like the one we're building here, the same principle has to apply. So our approach is to first train a general model that's training on as much data as we can find. So that is why in our Dyna-2 report, our model is trained over a million hours of human data as a starting point. And then given that strong foundation, what we found is we can quickly make the robot or make the model do many, many different tasks with a very minimal amount of fine-tuning. To just tell the model, "Here's the task I want it to do, and here's very few amount of demonstrations," and then the model
- 10:15 – 15:46
The Robotics Data Bottleneck
- JMJason Ma
can readily do bunch of the tasks that you have seen in the office. Yeah.
- AFAlessio Fanelli
Um, we actually have quite a few people who deal in data, who sell data.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Um, what is good data? What is bad data? What do you wanna buy? Uh-
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Uh, I think these are all o- open questions. Like obviously I think the, the top, um, important thing is egocentric data of humans doing tasks. Um, what else would you modify on that basic statement?
- JMJason Ma
Yeah, yeah. So I think data is the most important thing in robotics today because unlike language model, there is no internet, right? So every company that's building model, building hardware has to think about data. So there's, uh, roughly couple of different ways of operating here. One is that you can collect a lot of data yourself, which is what we are doing as well.
- AFAlessio Fanelli
You have employees here doing that. [laughs]
- JMJason Ma
I mean, the robots everywhere in the office, right? Uh, but, uh, if you think about how much data you need for robotic foundation model, I think the amount of data any individual company can collect by themselves is just nowhere near sufficient, right? It's kind of like, you know, you're trying to build an internet by just having Reddit or just having Facebook. Th- they're a great website, a lot of users, but it's not enough, right? So when we think about data, there has to be some sort of data acquisition happening, right? You buy data, you look at data that's already out there. And then there's many different tiers, right? Um, you mentioned data quality. I think data quality in robotics is a non-stationary problem, in the sense that what is high-quality data for your model changes over time. Like when your model doesn't know anything, like when it's zero, any data is useful, right? You can use a lot of video data. Uh, you know, in the case of egocentric data, I think it's actually a very good way of pre-training robots because- Human data is so diverse. Like, it covers every single physical activity you can possibly imagine, and that gives robot the model of such a strong prior on how things, how the world can evolve around a embodied actor human. But the issue with human data is that before your robot looks exactly like a human, there's gonna be some gap between human actions and the robot actions, right? So just by training on human data, uh, today, it's not very likely you can just execute robot actions directly. So you still need some amount of a- adaptation using robot's own data, right? But the robot's own data collected on a robot is gonna be much more expensive and much harder to collect. So the kind of recipe we're thinking about is, like, how do we take the largest source of data we have today, like egocentric data, to give the model a strong prior over everything physical manipulation can cover, and then combine that with a small amount of robotics data to make the robot, uh, you know, master the whatever task we're interested in. So data quality and getting data in robotics is really, I think, is a fascinating multifaceted problem. But the reality is that there's not a single winning strategy like, "Oh, here's one kind of data. I'm only gonna collect on that," which will give you a strong foundation model. You really need to think about, uh, this human data covers this part of robot cognition I need. Robot data covers low-level reflex dexterity the robot also needs, and put them into one model, and I think that's what gives a really adaptable, really strong model at the end.
- AFAlessio Fanelli
Yeah. I- I'll, I'll prompt you, uh, just for, again, for the people listening who will want to work with you.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Um, what, uh, d- what data vendors do you want? Uh, or this is your call for data. What-
- JMJason Ma
Yeah. Yeah, no, that's a great question. And, uh, yeah, so in our Dyna 2 blog at the end, we actually have a email address-
- AFAlessio Fanelli
Uh-huh
- JMJason Ma
... called datapartnerships@dyna.co.
- AFAlessio Fanelli
Okay.
- JMJason Ma
So if you're a data vendor that has a lot of high quality, let it be egocentric data, uh, robot data, video data, or anything you think could help robots, you know, send it to that email. Our team will take a look, and we'll get back to you.
- AFAlessio Fanelli
Yeah. But, like, at, at, at your stage, are you... You're, you're probably not buying any more egocentric data off the shelf, right? That you probably want to be like, "Here's what I need. You go-"
- SWswyx
Is there a niche that's lacking?
- AFAlessio Fanelli
"... You need a scale AI. You need a core for, like, doing this."
- JMJason Ma
I see.
- SWswyx
The-
- AFAlessio Fanelli
The custom data
- SWswyx
... people that don't work in robotics are probably thinking like, okay, some niche factory, like, you know-
- JMJason Ma
Oh
- SWswyx
... you showed putting tubes in. There's only so much data you'll find about that.
- JMJason Ma
Yeah.
- SWswyx
But, like, would a lot of that help you? [laughs]
- JMJason Ma
Yeah. No, I mean, if you really want to nail a task, having data from that task will certainly help you a lot, right? So for us, it's like we still think, you know, just a million hours of data is, is not enough. It's just the beginning. Like, we showed the first model on a million-hour scale, but we need more than that, right? So we're still looking at a lot of just raw, diverse data, uh, that we can get, but we're also thinking about how do we get more custom data either collected in-house or by our partner for, like, really, you know, specific niche tasks because I think you still need that to get to, you know, excellence that, you know, any model needs to actually be useful in the physical world, right? It's kinda like humans. You know, you know, you know, if, uh, you know, Sean or me, you know, we go work at Rosewood Hotel tomorrow, we're not gonna be able to make the bed the way they want, right? We need a lot of training. So that kind of data, like special data, it's kinda like what Marco is doing today for language model, I think will be very, very valuable for robotics as well.
- SWswyx
Yeah. So hotels, you know, strap a GoPro on your
- 15:46 – 18:47
Synthetic Data and Self-Improving World Models
- SWswyx
employees' heads and [laughs]
- AFAlessio Fanelli
And then [laughs] there's probably more of a-
- SWswyx
There's a, you know-
- AFAlessio Fanelli
They put themselves in their, in their wig.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Uh, so, so one thing that we discussed is, like, the, the closing this interview. Uh, and, uh, what's very interesting is, like, now you can basically fabricate environments on the fly.
- SWswyx
Yes.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Uh, uh, have you tried that with the, the frontier models? Um, what's your experience? Are, are they good enough?
- SWswyx
Are you using any world model robotic, like world model companies to generate these and train on them?
- JMJason Ma
Yeah. So, uh, I think generating synthetic environments, sy- synthetic data, simulation data for robotics is going to be one of the next frontiers, right? 'Cause today people have looked at egocentric data, robot data. You know, those are scaling. But I think synthetic data will be a new dimension of scaling. So going back to your question, like, yeah, we have been exploring, like, using frontier models to generate a synthetic, uh, environments for us on our data, on our environments that we care about. And surprisingly, I think now it's at a point where this technology is actually very much worth, like, going really, really deep into. So for some context, so, uh, in my PhD, I primarily focused on simulation-based training for robotics. Back then, simulation wasn't ready. But we had this paper called Eureka, which was the first paper taking like a large language model, at the time it's GPT-4, which is how to, like, self-generate, uh, environments, generate reward functions or reinforcement learning and just, like, do things in simulation. And at the end, I think, uh, we had this really nice demo of the robot spinning pencils, right? That was really cool at the time. But that kind of thing was still very, very limited at the time because back then the model didn't really have good vision capability, didn't have a innate 3D understanding, right? So it was basically just generating code and, like, using very limited feedback from text to try to come up with new simulation environments. But today, like, the vision capability of these frontier models are crazy. They can do, like, real to sim to CADing. There's 3D reconstruction, all of that.
- SWswyx
[laughs]
- JMJason Ma
So now it's at a tipping point where this technology could really accelerate robotics. And for us, right, we think about that, like, as, like, a way to accelerate our internal progress, right? Because we're building models, we're building hardware. So if a frontier model can give us more data that way, like generate really tailored data for ourselves, that's extremely useful for us. And another thing we've been thinking about is instead of using these models to generate, like, 3D environments, we have also used our own world model, right? The world-action model, uh, I was telling you about. Because the model not only generates action, it can also generate videos. We- the model can also generate its own training data. So that becomes a, you know, self-fulfilling, like, flywheel that I think will also be powerful. So overall, I'm just really excited about this direction because I think even though we have scaled to a million hours of data, we are deploying a bunch of robots. The robots are act- you know, are already starting to do very cool things. We're just at the
- 18:47 – 21:41
Real-Time Inference and Model Distillation
- JMJason Ma
beginning, just the tip of the iceberg for, like, what is there to come. Yeah.
- SWswyx
For those on the research side, we recently had Runway on the podcast.
- JMJason Ma
Yeah.
- SWswyx
One of their things is robotic world models. They're, um, shifting a little towards there, so I'll, you know-
- JMJason Ma
Yeah
- SWswyx
... nudge people to look at it. Um, one of the questions I have about you making your own models is what is the constraint you operate in? So if you draw a parallel to language models, you see, like, GPT-oSS 20B is-
- JMJason Ma
Yeah
- SWswyx
... very fit for your average MacBook.
- JMJason Ma
Yeah.
- SWswyx
Then there's a larger one that's fit for a single GPU. Um, do you work in the constraints of robot, uh, model on-
- JMJason Ma
Mm-hmm
- SWswyx
... on device, in cloud, a hybrid, and then how does that affect your scaling law of data?
- JMJason Ma
Yeah.
- SWswyx
Um, you know, a million hours is great, but, like-
- JMJason Ma
Yeah
- SWswyx
... do we just scale up parameters, data?
- JMJason Ma
Yeah, no, that's a great question. So I think what's really different about robots versus language model and other applications is that for, uh, coding agents, language model, it's okay if it's, it has latency, right?
- SWswyx
Yeah.
- JMJason Ma
But, like, you know, today, whenever I ask ChatGPT, it takes, like, a couple seconds before it spits out anything. But for robots, like, the inference latency has to be, like, very short, right? The robot needs to act. Like, it can't just hold a knife and not do anything for three second. That's gonna freak people out, right? [laughs]
- SWswyx
Yeah, we saw someone put their real hand under a knife.
- JMJason Ma
Our model's, uh, pretty smart, so that's okay. But, uh, in general, like, if you think about very dynamic tasks, like for example, you want your robot to like, you know, hold a, I don't know, a tennis racket and, you know, juggle the ball, right? It's this very dynamic, very reactive. You can't just like... The phys- physics, gravity doesn't wait for you, right? So for that reason, robotics model needs to be able to inference very fast, and the fastest way to do that is by serving on-prem, like, on device, right? Because if you call a model-
- SWswyx
Network latency
- JMJason Ma
... there's no guarantee.
- SWswyx
Self-drive a car or-
- JMJason Ma
There's no guarantee, right? Like, so for that reason, the model that gets deployed eventually has to be of reasonable size that can be served locally. But that doesn't mean that the initial model you train also has to be small, right? Like in language model-
- SWswyx
Can distill
- JMJason Ma
... people do distillation, right? Like the best way to make a small model very smart is by doing distillation, right? You train the biggest model, then you distill. Right. So I think in robotics, we're getting there, but I think that's what people are going to converge on. Yeah.
- SWswyx
How about, like, uh, inference? You have your own inference stack, or... A lot of this is architecture. You've pulled diffusion and whatnot. I'm guessing you can pull off of something. Anything you wanna share?
- JMJason Ma
Yeah. So we build our own inference stack for whatever model we deploy, because that's, like I was saying, like latency is so important, so owning the whole stack is extremely important for robotics, right? You want to optimize, like, the inference speed for the
- 21:41 – 26:58
Full-Stack Robotics: Hardware and Supply Chains
- JMJason Ma
kind of model you're running, and there's a lot of work in there, so that's what we do.
- SWswyx
Yeah. You're really full stack, like-
- JMJason Ma
[laughs]
- SWswyx
... everything from like, uh, you know, the actuator results, the way to the industrial design, ID.
- JMJason Ma
Yeah.
- SWswyx
Which I didn't really know as a acronym.
- JMJason Ma
Yeah.
- SWswyx
Uh-
- JMJason Ma
Well, I, I guess-
- SWswyx
Obviously to the models and data as well
- JMJason Ma
... uh, I'm curious, like are these your own actuators, or are they... How much of this is your own? Like of course the design, you know, it's not a humanoid, but-
- SWswyx
Yeah
- JMJason Ma
... actuators, like how, how far down the robotics stack? Yeah, so the way we think about, uh, what we need to own versus what we can buy is, like, if the, uh, input/output and, uh, of a component is very, very clear, and, like, there's already extremely good mature solution, then there's no point to building a house. Like for example, on this robot, uh, you see the LiDARs, right? So those, uh, black circle, those are LiDARs to help the robot detect obstacle, um, navigate surroundings. So that's a very mature technology that people have been using for now decades. It's on self-driving car. So we don't need to build LiDAR from scratch, right? Same goes with some cameras. But for anything that we think is extremely crucial for the whole system's performance at the end, our vision is to be able to own that and build that ourselves. So one thing is, uh, you mentioned actuator, so that's something we're working on, and we're going to own ourselves. The reason is that, like, the quality and how you design the actuator really matters for how well you can model the robot, right? Actuator is what drives the robot. To be, be able to model your robot very well in simulation or to do any kind of modeling, you need to select actuator that's very good, that, you know, can actually be easily modellable. So that's what we care about eventually. So we're gonna own that. Same thing with, you know, the robot's proportion. You know, this robot's proportion is, like, we measure against real people. Why do we do that? 'Cause we're pre-training on human data.
- SWswyx
Oh, yeah. [laughs]
- JMJason Ma
Best transfer, right? So there's a lot of these kind of, uh, decision that goes into hardware that's very much tied to, like... Because eventually, you know, it's hardware that carries intelligence. Intelligence is the extremely important thing. We want to make sure the hardware serves for that.
- SWswyx
Yeah, yeah. Uh, I'll just highlight, I, I don't know if this is relevant to you, but, uh, one of the most interesting robotics discussions I've had is someone who was very stressed out about actuators and specifically magnets. 'Cause apparently we don't make magnets in the US.
- JMJason Ma
No, we don't.
- SWswyx
And there's only two countries that make magnets.
- JMJason Ma
Uh, rare earth material.
- SWswyx
Yeah, right. I don't know what the, what, what's the chemical, what's the material. I don't know what the thing is, but-
- JMJason Ma
Yeah, yeah
- SWswyx
... uh, so yeah, if, if anyone's interested in that, that's, uh, apparently a, a bottleneck that we have.
- JMJason Ma
No, it's a bottleneck. Yes.
- SWswyx
So what do we do? [laughs]
- JMJason Ma
Yeah. I mean, I mean, it's a good question, right? I think, uh, there's a lot of, uh, exciting discussions on bringing manufacturing back- To the US, but, you know, for robots especially, that's actually much harder than, um, than what it seems. And I would say even now, there isn't a company in the US that's truly building everything from scratch in the US, right? Like, even when they say they assemble robot, they build whatever they're actually doing. In the US, they're still sourcing the raw materials from other places in the world, and I think that's something to really be aware of as a robot builder, right? Like, I think, uh, like, as a full stack robotics company, we realize just how long the supply chain is and how long the whole problem is. It's actually extremely hard. And I think these days, a lot of the discussion centers around a robot foundation model, model data. But I think hardware and just how hard it is to build good hardware is very underemphasized.
- AFAlessio Fanelli
Yeah.
- JMJason Ma
But I think that's gonna be extremely important.
- AFAlessio Fanelli
This is a, a platform, right?
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Actually-
- 26:58 – 31:31
Why Dyna Built a Wheeled Semi-Humanoid
- JMJason Ma
So if you wanna solve physical tasks, you wanna be able to optimize everything, and that's important.
- AFAlessio Fanelli
I mean, we have the robot. You gotta tell us about the design.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Um, your own custom platform. What do you call this? [chuckles]
- JMJason Ma
What do we call it?
- AFAlessio Fanelli
Yeah. So it's not humanoid. Um, it's very smooth, but what's the, what's the rotating arm?
- JMJason Ma
Yeah. So we think of this as generally like a semi-humanoid. So basically the upper body is, you know, uh, like a humanoid. It's got, you know, of course, shoulder, got arms. It also got, like, four degrees of freedom on the waist. So you can basically do all the upper body motion that a human can do. But it's a semi-humanoid because the lower body is wheels. All right, so why do we choose wheels? Because I think there is a difference. There is a, you know, optimal, like practicality versus performance trade-off. I think with legs, the energy con- the energy consumption will be a lot higher. It'll be a lot more unstable. You know, safety will be extremely, extremely hard to guarantee. So right now for legged robots, there is actually no guarantee that the robot will stay upright, balanced the entire time. I'm sure you guys have seen a lot of those, uh, robot dancing videos, like, like the, like the robot just fail.
- AFAlessio Fanelli
Seeing dancing, yeah.
- JMJason Ma
It's interesting.
- AFAlessio Fanelli
They, they all, they fall over very easily.
- JMJason Ma
No, but you also see the demos that you can tell are not production, where you can box the robot and you can push it over and it's very good. But that robot is just good at balance, right?
- AFAlessio Fanelli
Yeah.
- JMJason Ma
Like as a...
- AFAlessio Fanelli
I think, I think the more interesting thing that you guys have innovated, so like we have, uh, one of the prototypes, uh, which is the MU2.
- JMJason Ma
Yeah. [laughs]
- AFAlessio Fanelli
Back here.
- JMJason Ma
You can call it.
- AFAlessio Fanelli
This is, this is just like an up-down-
- JMJason Ma
Yeah
- AFAlessio Fanelli
... uh, axis and you've chosen to then move towards something like this, which, um, has the, the ability to lean.
- JMJason Ma
Mm.
- AFAlessio Fanelli
And we're, we're gonna show, uh, the laundry demo where you have to lean into machines.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Right? That's another degree of freedom that I don't even have the words for it. It's not pitch. It's not yaw.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
It's something else.
- JMJason Ma
Yeah. Yeah. So yeah, how, how did we think about that, right? Like, uh, so actually it's very used for exercise. Like, you know, I think sometimes, a couple years ago when I was exercising, I accidentally like, you know, like pulled a muscle.
- AFAlessio Fanelli
Mm.
- JMJason Ma
Right? I'm sure you guys have that experience too, right? Then the next few days, like you just couldn't move or like you pull or you slept in the wrong position, so your neck is very stiff. We have all experienced that. And then whenever you experience that, you kinda realize, wow, so many muscles, so many degrees of freedom that we use in our daily life that we take for granted. But when it's disabled, like it's very awkward for you to do some task that you feel are very natural, right? So that's also how we think about like when we design robots. When you have like a, this kind of design, which is this purely elevator could only go up and down, we realize a lot of the like human torso movement, like we use our like, you know, waist back a lot for like day-to-day physical activity, right? Even like, if I wanna get this bottle, like I'm not just moving my arms. It's actually very strange for me if I, I have to intentionally think about fixing my upper body. I would always like do this, right?
- AFAlessio Fanelli
Mm.
- 31:31 – 33:55
Teaching Robots New Tasks and Evaluating Performance
- JMJason Ma
seeing that like the transfer is actually there and is actually very helpful. Talk about the eval side of this. Yeah. So you can have predictions, you can have it do stuff, and you know, every now and then we see it miss stuff. You test on very niche things as well, so like filling a pill bottle. Yeah. Um, how long does that process take? How do you think about the aspect of actually deploying on a new task, evaling it, iterating? What does that look like? Yeah. So if we wanna learn a new task, uh, you know, we basically do a little bit of robot data collection. Then the model, these- What is a little bit like? Yeah, like couple hours. Couple hours. Yeah. We're, in some of the tasks, like just tens of minutes, right? Yeah. And then- The chopping one was 45 minutes. Yeah. The open the fridge was on- And that's an interview. That's like an onboarding task for one of your engineers. Yeah, yeah, yeah. [laughs] So, uh- Fun, fun thing if you wanna work here, you know? Yeah. Find the task, have the robot do it. Yeah, yeah. Pitch me a new task, and I'll come to the office, uh, you know, collect some data, train a model, and see what happens. Tie your shoes. [laughs] Julie is tying. Yeah, that'll be cool. But, uh, yeah, so you know, a little, a small amount of data collected, train a model, then we just deploy the model in the real world. And then we look at the performance, then- To, to fill in collected means like you are, you have a person actually operating- Yeah, operating ... remote control- Yes ... doing the task for a bit. Yeah. And then you train? Yeah. And it does okay. You improve. Yeah. How do you measure? Yeah. Yeah. So when you're training the model, there's, uh, you know, certain metrics you can look at, right? Just like training a model, it converges nicely. Now after that, for robotics today, we just run the model in the real world, right? On the, you know, same setup that the data is collected to see how the model does. Sometimes the model just works great out of the box, right? You don't really need to do much more. But when the model's, you know, like maybe like 70%, 80% there, then you see how the model behaves. You figure out what it's not good at, then you can like collect a bit more data, then iterate, right? I think of this as like the robotics version of like reinforcement learning from human feedback, right? The robot rolls out. You can score it, good, bad, s- uh, somewhat good. And then that data, the robot's only experience can be used to train the r- model, and you can also just collect more data yourself to train a model, right? Eventually, you know, our goal is like, you know, this robot
- 33:55 – 38:02
Autonomous Workflows and Hardware Limitations
- JMJason Ma
is gonna be our platform. Anyone should be able to, uh, train tasks on it. I wanna get a bit of a prediction and like timeline to where that goes. Right now the tasks are, um, somewhat simple, like- Yeah ... take laundry out. It's a towel, you can't crush it. Um, chop some food. Yeah. There's also the side of like, okay, set up this filming studio, right? Yeah. Move stuff around, um, bring a table, set up lights. I think manage my data center. Yeah, yeah. Plug in stuff. Um- And, and also, uh, re- like, uh, recharge another robot. Yeah, yeah. When, when do we see the factory or like, w- what do we need to scale up? Um, is it model? Is it everything? It seems like your hardware is set for that, but where do we see it go to a point where I can go from instead of just cut one thing to, you know, okay, here's a recipe- Yeah ... like full send or- Yeah. Yeah. Yeah. I would say to get to truly general purpose robots, uh, like, like basically a robot or a model that does anything, like- Not, not, not, not that. Like okay, so to keep it somewhat grounded on a technical level, like- Yeah ... when do we start to see us pushing that feedback, right? So- See ... instead of very simple chopping- Yeah ... uh, maybe a five to ten minute sequence, right? Yeah. Where now the eval step, it can break at many more points. Instead of just fill one pill bottle- Yeah ... uh, 10 different pills all in a sequence, right? Yeah. What's limiting us? What are we pushing on? Yeah, I think, yeah, I think couple of thing. One is that like, uh, the laundry demo that, uh, you have seen, yeah, it's gonna be like one hour to finish a whole load, right? So the longest duration that a robot can do like a full job is already in the hour range. Nice. Right? But, you know, to be able to do that for any task, you know, I think the bottleneck is coming back to data and, uh, hardware, right? So for some of the tasks, you know, for example, setting up the space, you know, you might need more than like a, you know, the parallel jaw gripper, right? You might need something more dexterous, like a hand, like a human hand, right? But then if you wanna train a model on the hand, then you're bottlenecked by the hardware stability. Like hand today, the best hands, dexterous hands, are still not, uh, durable enough. They overheat pretty quickly. Right. So that becomes a bottleneck, right? Like you, we, you know, you quite literally have to put a, a, you know, ice bag- Ice [laughs] To cool- We're not- Human ... we're not just auto researching, make better model. We have, we have hardware. Yeah. So hardware limitation is a big thing. But then if hardware is limited, then that will slow down everything, right? Because to know that you're making progress on any task, you eventually have to eval on the hardware, right? So then without the eval being very like spontaneous, very fast, that also limits your research progress, right? Because then how do we collect the right data, right? I think, uh, how do we collect the right data to do extremely complex, like setting up this, uh, studio, right? Because you can imagine- Like today, most of the data is, like, vision-based, right? It's like video streams and, like, you know, some, uh, uh, encoding of your hand motion or the robot's, uh, joints. But humans have a lot more sensors. You know, we hear things. We feel things.
- SWswyx
Smell.
- JMJason Ma
Yeah, you smell things, right? So we subconsciously use all of those to help us be so good at manipulating the physical world. But today, like, if we're not capturing that at all, then even when we scale to millions of hours, you might just be missing critical dimensions of the task, so you'll never be able to solve the task, right? So I think those are, like, what I consider, like, you know, what's dragging out the timeline. But for tasks or for workflows that you can collect data today to high fidelity, then I don't ne- I don't necessarily think there's a fundamental bottleneck, right? Just like what we are demonstrating, uh, to the public this week. You know, we have a physical agent that can do the entire laundry workflow, right? As a human would, right? Because for that task, we don't really have like, oh, you actually need more than a parallel jaw grip, or you actually need some sensing modality that's just not available today, right? So I do think the biggest
- 38:02 – 43:03
Real-World Deployments at Best Western and Din Tai Fung
- JMJason Ma
bottleneck to have a much more general thing is still hardware and data.
- SWswyx
Yeah. I want to take this chance to ta- uh, let you brag about your customers and partners.
- JMJason Ma
Yeah.
- SWswyx
Uh, you have some announced, uh, relationships. Um, we can start with Best Western and then maybe some of, uh, some of the others that you're-
- JMJason Ma
Yeah
- SWswyx
... that you're proud of.
- JMJason Ma
Yeah.
- SWswyx
Cool stories, anything like that.
- JMJason Ma
Yeah. So we have a deployment blog post, uh, on our website that goes into much more detail. I really recommend watching the video because I think the video is-
- SWswyx
Which one? The Best Western one?
- JMJason Ma
Uh, no. We have, like, a, one video for all the customers we're working with.
- SWswyx
Okay.
- JMJason Ma
It's a really, really nice video. It shows what we're doing and, uh, yeah, I think it's great. But yeah, we have customers, uh, you know, Best Western, that's in the hotel space. We fold a lot of, uh, laundry, like, uh, you know, towels for them. Every hotel needs it. Every gym needs it. Every fitness center needs it. Uh, but-
- SWswyx
What's the deployment so far? Uh, how... Like, where can I go to... Which Best Western do I go to to see it?
- JMJason Ma
Yeah. So if you want to go somewhere, I actually recommend going to our other deployment, Din Tai Fung. Have you guys been to-
- SWswyx
Yes. Yes.
- JMJason Ma
Yeah. You know the best-
- SWswyx
Very thin wrappers.
- JMJason Ma
Yeah. Very-
- SWswyx
We should have filmed there. [laughs]
- JMJason Ma
Yeah. No, we should have. [laughs] We should have just filmed there.
- SWswyx
B-roll.
- JMJason Ma
Yeah. But, uh, in Din Tai Fung, like, uh, if you go to Din Tai Fung or other like, uh, you know, somewhat upscale restaurant, there's napkins on your table, right? So our robots, our robots today fold thousands of napkins for, uh, many, many different locations, many different Din Tai Fung locations in the United States. So that's actually our biggest scaled customer, right? So I wanna make a distinction here. Uh, today there's a lot of robotics companies that's talking about deployments, talking about pilots, but what's really hard is not putting a robot in one location or doing a pilot. What's really hard is that the customers believe or validate that your robot creates enough value that they want to deploy your robots across many of their sites. So that's what's happened for us with Din Tai Fung, which to the best of my knowledge, hasn't really happened for other companies doing similar, like, uh, general purpose robots. So at Din Tai Fung, our robots fold thousands of napkins per day. It's... The quality standard is extremely high, you know, because they want like, okay, uh, the second fold, or the seam of the napkin is only, like, a couple centimeters below the, uh, I don't even know how you call it, the, the, the pocket, so you can stick the chopsticks in. So it's actually a very fine grain, high precision task. I can show you.
- SWswyx
At some level, that's what robotics are good for, robots are good for too, right? Um, humans are not the most consistent-
- JMJason Ma
Yes
- SWswyx
... especially in small details, whereas a robot can somewhat do it perfectly.
- JMJason Ma
That-
- SWswyx
It's interesting that's the example you bring for general purpose robotics versus, uh, uh, at some level I think people would assume-
- JMJason Ma
Yeah
- SWswyx
... you can have a specialized robot that will fold napkins perfectly to the exact, you know, millimeter as opposed to something like this, but...
- 43:03 – 48:09
The Bitter Lesson of Robotics
- SWswyx
same thing.
- AFAlessio Fanelli
Yeah.
- SWswyx
If you don't know, you don't know.
- AFAlessio Fanelli
Yeah.
- SWswyx
But, you know, that's the, that's the funny-
- AFAlessio Fanelli
I mean, the, the, the term that comes to mind is like, kind of like bitter lesson for robotics, right? You, uh, instead of the specialized robots, which is a lot of, uh, industrial engineering, you build a generalized platform that can-
- SWswyx
Yeah
- AFAlessio Fanelli
... uh, do a last mile fine-tuning and, and-
- SWswyx
That's right
- AFAlessio Fanelli
... and do all those things.
- SWswyx
Like give the expert add-on outside of your what is, what is more bitter, bitter lesson robotics? [laughs]
- JMJason Ma
Yeah, I think there's several bitter lessons for robotics. One is that robotics is going to scale just like any other domains that machine learning has solved, right? So what does that mean? That means have as much data as possible. But in robotics, the one thing that's a lot harder is, like, what is the right data is much harder to figure out than other domains, right? I mean, language is text. Uh, in video games, just video game data. But for robotics, like what sensors you choose determine what kind of data you have, right? But if you have the right s- set of sensors and just scale that data, I think you'll get predictable return on model performance. So in that way, robotics isn't so different. So that's the bitter lesson. As a corollary of that, is that, like, there's a lot of research in robotics that's coming out with, like, very specific priors. Like for example, we need to make the robot see objects as they are. Draw some boxes. That will give robot better perception, right? But I think the bitter lesson here is that just keep the algorithm simpler, and scale the right kind of data, then generalization, better behavior, will emerge. So that's not so different. And the second... I think that's the main bitter lesson, which is like, at the end of the day, scaling will work. But in robotics, to scale and how to scale, just much harder, 'cause there is no internet. You have to come out with your hardware, come out with the sensors, get all of those right, and then scale. Then I think, uh-
- SWswyx
I mean, there's the argument that, you know, the world is very made for humans. It should be a human, right? As hard of a hardware problem that is, think 10, 20, 100, 200 years down the line, can we make a really efficient human? But you have to be practical with what we have now, right?
- JMJason Ma
Yeah. Yeah. So the way I think about it is like, yes, eventually the robot should... If we wanna build a general purpose robot, right? Like a machine that can be adapted to as many use cases as possible, I think human form factor will be the best one, just because that's what we have been doing, uh, for ages, right? But I think the best way to actually build a humanoid robot is to take a iterative approach, because robotic is so hard. The, you know, the whole problem, the whole stack is so long, it's really hard to know a priori what's the right decision to make, right? And because hardware decision is much more expensive and much more costly, just 'cause hardware iteration is a lot slower, you actually want to take a incremental iterative approach, not going all the way to the final form factor. Because the chances that the thing you build actually doesn't last, it's not good enough, so you just end up scratching it anyway. You know? Like, it's public news, like, you know, Optimus, you know, already gone through many, many iterations, and they had to scratch their designs, right? And I think by iterating with a semi-humanoid form factor and actually putting them out to real customers, or, and just getting models really deployed, you get that feedback loop that you wouldn't get if you're just trying to build, like, you know, warehouse humanoid, and you spend like 5, 10 years building it. I think the iterative approach will actually get us closer to the final robot faster.
- SWswyx
And I will say, you know, for those listening and not watching, it's probably the smoothest robot we've seen. Like-
- JMJason Ma
Thank you
- SWswyx
... every humanoid you've seen is jerky.
- JMJason Ma
[laughs]
- SWswyx
And even humans don't walk as smooth as this thing probably.
- AFAlessio Fanelli
Because, because, yeah, the model is-
- SWswyx
Uh
- AFAlessio Fanelli
... can, can move every single m- motor at once.
- JMJason Ma
Yeah.
- SWswyx
You're more than two cohort. Yeah.
- JMJason Ma
Yeah. Yeah, and that's actually a very hard problem, because if you th- just think about, like, uh, the model, you know, it's outputting action. There's the action, you know, dimension, right? The bigger the dimension is, the harder to fit, uh, the model to the data. But here, the action space is quite large. But, uh, uh, we did very good job to make sure the model fits well and gives you a smooth-
- AFAlessio Fanelli
Can you bound the order of, of magnitude of the action space, or is that...
- JMJason Ma
Yeah.
- AFAlessio Fanelli
All right.
- JMJason Ma
So it's in the orders of like, uh, you know, about 100 action dimensions.
- AFAlessio Fanelli
Yeah.
- 48:09 – 51:03
Agentic Harnesses for Physical AI
- AFAlessio Fanelli
where it is, there's agentic planning and tool hauling and all those things.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
So we can show that-
- SWswyx
I mean-
- AFAlessio Fanelli
... for listeners
- SWswyx
... I think that's an interesting side after the model.
- JMJason Ma
Yeah.
- SWswyx
How do you wrap it around into agentic harness? Like, what is-
- JMJason Ma
Cloud code
- SWswyx
... final deployment? [laughs] Cloud code, a UI. You have a lot of cool UIs. Uh-
- JMJason Ma
Yeah
- SWswyx
... but you know, what, what goes into wrapping model output towards final thing? Is this custom per use case? Is it you have your own SDK?
- JMJason Ma
Yeah, yeah. So I mean, eventually we will have our own SDK, but, uh, the way, the way we think about is we want a general, like, harness layer for robotics foundation model. Uh, today that is our own model, right? The way I think about is like, for any like- Real task, long task in the physical world. There is a lot of, like, a high-level decision-making you have to do, right? It's not just like robot, you know, only reasoning about the physical world in the action, continuous trajectories, right? 'Cause that's a very, like, low-level, the thing you're actually executing in the physical world, but that's not the intent that, you know, like if- when we are doing decision-making, we think about steps of a recipe. We think about, "Oh, like I finished this, I need to do that. While I'm waiting for this part to finish, I need to go somewhere else to do something else." There's a lot of high-level cognition decision-making that's happening. And I think for robots to handle even, you know, the laundry task we are showing you, it has to have that capability because, you know, it needs to reason that, okay, the washer is running, so I should go do something else while the washer is finishing. But now the washer is done, I should take, I should take the towels from the washer and put it in dryer immediately, right? Because if not, then that becomes a bottleneck in the whole workflow, right? So there is a lot of reasoning that needs to happen for robotics, and I think in theory, you could just, you know, try to learn that with a, you know, low-level model that's outputting actions, but it'd be extremely inefficient. So that is why having a hierarchical system, like putting a harness, like agent- agentic system around our world-to-action model is what allow the model to compose different capabilities, different skills to solve a much longer horizon task. It's very much like in language model, right? Like two years ago, three years ago, the model was only good at autocomplete. Like humans have to like tap, tap, tap to get the model to keep spitting out code. But today, just like agentic system just carries the end-to-end task, and that's... And, and I think that's important because I think today in robotics, people are still thinking about autonomy at the very task level, like a one-minute, 30-second task. But really, the value is about tasks or like workflows that go on, like unsupervised for hours, and that's what we're building towards. And that's why I think thinking about the robotics problem, not just as a modeling problem, but rather as a system problem, you need good agent, good harness engineer, good model, good hardware, everything.
- 51:03 – 53:58
GPT-6, Coding Agents, and AI-Native Robotics
- JMJason Ma
It's like how we think general purpose robots ought to be solved.
- SWswyx
Do you guys see benefits from language model coding tools getting better? So new Opus 55, new Astra. Um, do you have language models in your agent harness?
- JMJason Ma
Yeah.
- SWswyx
How does AGI being developed in language model space affect you guys? [laughs]
- JMJason Ma
No, no. Yeah, no, absolutely. Like it affects us very deeply in many ways. One is that like we are extremely AI-native company, right? Like coding agents, all of that is a very fundamental part of our work, and we're very welcoming of that. Like whenever there's a new model, we're the first one to test it, right? Like we've been... Like when GPT-6 came out, we had like a company-wide hackathon the same week, and the thing that our engineers built was incredible, and many of them are-- many of the features we built is already in main, in production, right? So we really like using coding agents to accelerate our work.
- AFAlessio Fanelli
Uh, just for the people interested, uh, anything surprising in GPT-6 that you found?
- JMJason Ma
Yeah, I think-
- AFAlessio Fanelli
On the code
- JMJason Ma
... uh, it's about the ability for it to like construct virtual environments, right? Turns-
- AFAlessio Fanelli
I-
- JMJason Ma
... to simulation. We talked-
- AFAlessio Fanelli
[laughs]
- JMJason Ma
... that, uh, 20 minutes ago. I think that's really exciting.
- SWswyx
It's interesting. There were so many like CAD Zillo 2 3D-ish.
- AFAlessio Fanelli
He lost the hackathon to the-- a CAD, uh-
- SWswyx
I won the-
- AFAlessio Fanelli
... ledger
- SWswyx
... I did the CAD bullshit [laughs] other way, other way.
- AFAlessio Fanelli
No. Oh.
- SWswyx
That was-- I won a GPT hackathon just for fun. I was like-
- JMJason Ma
Yeah
- SWswyx
... I know nothing CAD.
- JMJason Ma
Yeah.
- SWswyx
Um, might as well play around with it 'cause every demo is doing it. I'm like, "It's gonna be garbage." And I'm like, "Well, it's kinda grounded, it's kinda good, but-"
- JMJason Ma
Yeah.
- SWswyx
"... it's just hackathon stuff." [laughs]
- JMJason Ma
Yeah. Um-
- AFAlessio Fanelli
Cool. I, I, I, I was gonna round out, uh, some of the use cases and then go into a little bit of company culture and your personal background.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Yeah.
- 53:58 – 1:01:13
World-Action Models vs. VLAs
- JMJason Ma
our goal next year is like overall just deploying hundreds of robots.
- SWswyx
Uh, my part two on the last technical question is, how do you see the rest of the industry? So fill in for people, what other approaches are people taking to world models? Why are they wrong? Um, what is different but good about their approaches as well? Just what is the landscape of robotic world models-
- JMJason Ma
Yeah
- SWswyx
... in this space, and why-- how have you come to your current?
- JMJason Ma
Yeah. No, that's a great question. So I think robotics is the most exciting space today, and there are just so many, uh, uh, you know, amazing companies that I truly respect. You know, they've been doing great frontier work like us, and, uh, it's very exciting. And I think what we are seeing today is that there's still many different approaches to robotics. Like we believe that hardware needs to be built in-house. We need to own the whole, uh, chain. Uh, we believe that scaling human data is extremely important. But there are companies that's betting on just scaling robots data or scaling some sort of intermediate, like, data collection devices, right? All of them are in play, and, uh, I think it's different approaches. We'll just see what happens. I think right now, uh, it's still a bit early to tell, but we feel pretty good about the way we're building things just because, although we as a company culture-wise, mission-wise, we're not here to just build a model. We're here to truly get robots deployed. So we deploy as much as possible, work on the hard problems, so we identify problems much earlier on. So then I think that really helps us identify what is the true bottleneck and then solve it. And I think for, uh, folks, for people who are only working on one part of the problem, it's kinda like fog of war. You don't really see the whole picture, and you might be building something that's extremely cool technologically, but may not be what people need or want at the end, right? I think there's a whole graveyard of robotics company, and I think all of them had very shiny, very cool robots. But, uh, you know, just figuring out what people actually want is actually... It's very easy to say, you know, like, "Build something people want." You know, everyone says that, but it's actually really hard to execute a robotics company.
- SWswyx
On the technical side, you have your world-action model. Is that standard across industry? Is there other-
- JMJason Ma
Yeah
- SWswyx
... directions people are taking?
- JMJason Ma
Yeah, I would say that's also not standard. So we're building world-action model on top of world model. So there is many companies building world model. You know, you mentioned Runway there, also getting to adapt, adapt-
- SWswyx
Video world models
- JMJason Ma
... right. Uh, there are also companies building, uh, vision-language-action model.
- SWswyx
VLA.
- JMJason Ma
So that's like taking vision-language models and adapting them to robotics. Uh, so I think there's many different ways to go about it. But I think at the end of the day, like, what we call the models is less important than what we, like, believe the model needs to be able to do at a first principle level. So I wanna explain that a bit. I think for robotics, the most important thing is that you need to be able to predict the future, right? Like robotics, any physical system is fundamentally about based on the information you have at the present moment and before, what should you do next? So it's really about predicting what comes next. So predicting robot actions is like predicting what the robot should do next. But if you think about it, robot is just a part of the world, right? Robot influences the world, but the world also influences how the robot should behave in that environment. So if you think about it this way, then the maximal, the most first principle thing you can do is just predict as much of the world that you can possibly predict, right? So that's why we believe predicting the future is extremely important. And today, the instantiation is world-action model. And VLA only predicts robot actions where some more abstract representation of the world. But whenever you're predicting abstract representation of the world, you're encoding your own bias into what's important to predict. And the bitter lesson tells you, let's remove human biases. Let's just take the most raw representation, the camera inputs, and just predict all of that, right? So that's how I think about, uh, modeling for robotics today.
- AFAlessio Fanelli
So just to press you a little bit on this, you're, you're basically you're saying your approach is better than a VLA, right? Because you're, you're, you're, you're emphasizing prediction. I fully buy that. I... Like, you've sold me on it.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
If you were wrong, what is the most likely number two [laughs]
- JMJason Ma
[laughs] So-
- AFAlessio Fanelli
That someone else is doing?
- SWswyx
Interesting question. [laughs]
- JMJason Ma
Yeah. So yeah, there is a caveat in what I've said before, and I think that's where I could be wrong, right? I said that you need to predict the world as much as possible, right? But what that doesn't say is that, does your sensor have everything about the world that you need to predict? I think I touched on this a little bit, right? Like, if you're training a world model that's purely based on videos, I think it'll be a very powerful world model, but it may not be a sufficiently rich enough world model because you're missing critical aspects of the world, like sound, like touch, right? So I think what, where we could be wrong is like, okay, I have the best world model on vision, vision-based world model, but tomorrow someone else comes out with a vision plus tactile world model that's just like 100X more data efficient just because, you know, vision is a very high information or, like, a highly redundant, you know, input because, you know, every other pixel that they don't change that much. You can see, you can understand the world without perfect vision. But tactile, you know, other modality might be much denser in information, right? So maybe having that, predicting that would just allow you to use much less vision, so you can get to the same performance-
- SWswyx
Ah. Uh-huh
- JMJason Ma
... with much less compute, with much less data, right? But-
- AFAlessio Fanelli
I think you will be able to adapt very easily.
- JMJason Ma
No, exactly.
- AFAlessio Fanelli
True. [laughs]
- JMJason Ma
The reason why-
- SWswyx
There's also a level of how much data-
- JMJason Ma
Right. So the reason why I bring this up is because I think if vision tactile world model or vision tac-audio, whatever world model, is the final thing that solves robotics, then I think robotics companies will be able to do that better than video generation companies that's trying to pivot to robotics. The reason is, like, to understand sensing is actually very complicated. Like, we've been building that since the company started, so we know the bottleneck. We know how to combine them. But for, if you're just trying to adapt a video model to robotics action, then like, you know, if we were wrong, if, you know, video model is not enough, then, like, you don't really have the know-how to get tactile to work very well, but we have that.
- AFAlessio Fanelli
This is a side question, but I, I think it's a quick one. Uh, heat seems very important. Is infrared just another color to you, or is that a-
- JMJason Ma
Yeah. You could be another-
- 1:01:13 – 1:06:16
From AlphaGo to Founding Dyna Robotics
- JMJason Ma
a lot you can do without understanding, but of course-
- AFAlessio Fanelli
It's just not in their distribution.
- JMJason Ma
Yeah, yeah. [laughs]
- AFAlessio Fanelli
Uh, so, so I wanted to bring it a little bit to your personal story or your background. Um, we both went to Penn.
- JMJason Ma
[laughs]
- AFAlessio Fanelli
Uh, I did not-
- JMJason Ma
I just found out today.
- AFAlessio Fanelli
Uh, yes.
- JMJason Ma
Crazy.
- AFAlessio Fanelli
Yes. Uh, I, I, you know, my, my, my big regret is I didn't join the self-dri- the DARPA challenge.
- JMJason Ma
Okay.
- AFAlessio Fanelli
Which, uh, then I could have been, like, early-
- JMJason Ma
Yeah
- AFAlessio Fanelli
... Waymo or something. Um, you were at GRASP.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
And then you, you've also worked at, uh, Google and NVIDIA. What are the different focuses of, let's say, academia versus Google versus NVIDIA? You've seen all these places, right?
- JMJason Ma
Yeah.
- AFAlessio Fanelli
And then, like, that obviously caused you to say, like, "Screw all that, I'm gonna do my own thing," right?
- JMJason Ma
[laughs] Yeah.
- AFAlessio Fanelli
You could have joined OpenAI's robotics lab.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Uh, so can you just take us through chronologically, but also, like, uh, one thing you learned from each?
- JMJason Ma
Yeah. So chronologically, yeah, maybe it's useful to talk about how I got into AI in the beginning, right? So, uh, as a kid, so I spent time in Japan and China. I grew up playing Go, you know, the Go in AlphaGo. So when AlphaGo came out, I was a senior in high school, and I was mind-blown because I've been playing this game for many, many years. It's so complicated. Like, I think before AlphaGo came out, people were like, "Okay, Go is one of the game that AI would never solve." The search space is, like, larger than the total number of atoms in the world. How could you possibly search for solutions in that? Then like, you know, RL, reinforcement learning, just, like, cracked it, right? And I think that was how I got into AI initially. Like, I was just really interested in, like, models or systems, agents that can self-improve, right? So when I got to Penn as a PhD student, I just really wanted to work on reinforcement learning, so I'm co-advised. Both my advisors were interested in, uh, RL at the time. One of my advisor was more in robotics. He did a postdoc at Berkeley robotics doc. So that's how I got into robots, uh, at Penn. You know, we have GRASP Lab, I think one of the best places to do robotics, and, uh, so that's how I got into doing RL for robotics in my PhD. And then, yeah, I did a bunch of research thinking about how to connect reinforcement learning, uh, with human data, how to do reinforcement learning, uh, theory in a very rigorous way. And then I went to NVIDIA. That's when GPT-4 came out. At the time, we were very interested in just solving a pen-spinning task. I don't know why. Like, uh-
- AFAlessio Fanelli
It's the dexter- it's the dexterity limit, right?
- JMJason Ma
I mean, I think-
- AFAlessio Fanelli
That you can solve Rubik's Cubes and then you-
- JMJason Ma
The time I was working with, uh-
- AFAlessio Fanelli
Jim Fan
- JMJason Ma
... Jim Fan at UCO, which you guys probably also know. Like, yeah, we were just like, "Okay, let's try to do dexterous pen spinning." The initial idea was, like, learning it from human videos, right? We were trying to watch humans do it. But then I realized, okay, today, we don't have a dexterous hand that can spin pencil at all, so how would I even be able to evaluate the algorithm even if learning from human worked at all? So I said, "Okay, we need to first try this in simulation." Then as soon as I tried to set up a simulated environment for a robot hand with a pen, I immediately saw what the issue is. The issue is, like, like, if I had to come up with a simulation environment, the reward function for reinforcement learning to solve pen spinning, that's impossible. Like, I wouldn't be able to come up with all the terms that you need to get RL to learn the right behavior. And then I, I was just thinking, "What if a model that can write code can do that?" That's what became the Eureka project, right? We basically... Like, in my opinion, we had the first, like, recursive self-improving robotic agent in simulation because we really, like, used that to solve a new task that no one has demonstrated before. And that, that, that allowed me to see the potential of using language model, using coding models to really accelerate robotics. So then I went to Google DeepMind because at the time, you know, I mean, even today, they have one of the frontier models in Gemini, right? So I was looking at how to use Gemini for robotics. But eventually I realized, like, I had a lot of fun doing academic research, but there was always a question in the back of my mind, like, "Does any of this matter?" You know? Like, when you're working on this kind of thing in the lab, like, you know, you ask me, "How do we come up with so many interesting, fun tasks?" You're basically doing your own eval. You're grading your own homework. Like, I wanted something more real. I wanted something that could actually be used by, uh, millions of people in the world, and I felt 2024 was when we started Dyna, like end of 2024. That was the right time, that we can take some of these technology to actually putting them out in the real world. So that's why, uh, I started Dyna with Lynn in New York. And, uh, fast-forward two years, we built our platform, our model. Our robots are in Din Tai Fung. Go check it out and, uh-
- AFAlessio Fanelli
Go eat. [laughs]
- 1:06:16 – 1:21:04
Dyna's Research Culture and the Future of Robotics
- JMJason Ma
seeing internally, like, this might change, and, uh, that's very exciting.
- AFAlessio Fanelli
Yeah. And, uh, so n- then also let's talk about company culture.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
Um, what principles you're, you're building the company with. Uh, obviously, like, the, the, you're, you're responsible for the science-
- JMJason Ma
Yeah
- AFAlessio Fanelli
... and the, the research, uh, agenda. Uh, and it, it just seems like when we meet the engineers, they're, they're very engaged and then they have a lot of ownership.
- JMJason Ma
Yeah.
- AFAlessio Fanelli
They have, they have a lot of motivations.
- JMJason Ma
I'll say it's, it's interesting to see how many side projects people just work on.
- AFAlessio Fanelli
[laughs]
- JMJason Ma
So some of the demos we've seen, it's just the engineers that work on it were like-
- AFAlessio Fanelli
Yeah
- JMJason Ma
... "Yeah, this seems fun. Might as well do it."
- AFAlessio Fanelli
Yeah.
- JMJason Ma
Um, to your point on fun evals, right here, like two feet behind the camera-
- AFAlessio Fanelli
Can't talk about that.
- JMJason Ma
No, I mean, the thing is just the e-
- AFAlessio Fanelli
Can you not talk?
- JMJason Ma
The eval is just-
- AFAlessio Fanelli
[laughs]
- JMJason Ma
... there's highlighters, four highlighters being rubber banded together, right? It's like, who comes up with, "Ah, yes, we need to test putting rubber bands on highlighters." It's just-
- AFAlessio Fanelli
Stationery-ish
- JMJason Ma
... fun stuff. People are very, uh, you know-
- SWswyx
Creative here. It's a beautiful space
- JMJason Ma
Yeah, I think culture... Yeah, so let's first talk about, you know, how we run the company, right? Like, we think of ourself as like a, like a, like a eventually like a research plus product company.
- SWswyx
Mm.
- JMJason Ma
It's kinda like OpenAI, where you do frontier research, but eventually all of them make their way into Codex through ChatGPT. That's how we think about it, too. We're not here to just do open-ended, like whatever thing that interests us at the time. We're always like product-driven. Like, we wanna deploy our robots, our platform to a new customer. What are the technical problems we have to solve, right? We have to solve generalization. We have to solve whole body control. We have to solve how to adapt quickly. We have to solve how to make the robots follow language. We have to solve dexterity. So once all those problems are very clear, then like... I mean, our engineers are very capable. They're very smart, so they just create instances of the task. They create tasks that will stress test those capabilities, right? Like the j- the grabbing a drink task is just how we test language following, right? You can just specify what you want. If the model follows intent, follows instruction, it should be able to do it. So that's how we think about like at the top level, we identify what are the things we have to solve, then we have a lot of bottom sub-project to like actually drive execution to solve them. And, uh, you know, you hire the right people, and everyone here is super passionate about robots, you know? They've been doing this for a long time. They love it. And we just create a culture where it just naturally happens, and, uh, it's, it's a really fun place to work.
- SWswyx
What's the split between like, uh, technical research software versus hardware-
- JMJason Ma
Yeah
- SWswyx
... side of the company?
Episode duration: 1:21:05
In the uListen app, the transcript lights up word by word as you listen. Get uListen
Transcript of episode BHiYomkfoGQ
