Inside the Robot Data Factory: Mecka's Josh Gao on Sensorizing Humans to Train Robots

1. 0:00 – 1:46

   Introduction

   1. JGJosh Gao0:00

      There's a, an idea in science fiction that is when robotics and AI has ultimately fully generalized and solved all tasks that humans can do, the final frontier is gonna be organic. Final frontier is gonna be biological. And you see this in, like, Asimov, you see this in Ender's Game, you could e- even in Hitchhiker's at the very end, the planet ultimately was this giant organic computer trying to process and understand what the meaning of life was or what the question was.

   2. SHSonya Huang0:27

      [upbeat music] Today, I'm delighted to have Josh from Mecka on the show. Josh is founder and CEO of Mecka AI, which is building a data and deployment company for robotics. I, and I think the whole ecosystem, think that robotics is about to enter a golden age. And unlike LLMs where we had so much free human data from the internet, robotics is fundamentally data constrained. And I think your position in the ecosystem as the leading data factory, uh, to bring robotic data and physical data in distribution for models is just a really, really interesting perch. So I'm excited to use this conversation today to chat about how does egocentric data and data collection work in robotics, what are the actual operations and economics of operating a data factory, and your, your overall thoughts on robotics research and deployment. Thanks for joining the show.

   3. JGJosh Gao1:36

      Amazing. Thanks for having me.

   4. SHSonya Huang1:37

      Awesome. Uh, maybe to start off, for those who aren't familiar, what is Mecka and how are you looking to impact the world of robotics?

   5. JGJosh Gao1:44

      So Mecka is,

2. 1:46 – 2:45

   What Mecka is: pixels, actions and logic

   1. JGJosh Gao1:46

      like you said, the data and deployment company for physical AI. What that means is today we create, we process, and ultimately we license a lot of the data that goes into training these robotics foundation models. The big problem across robotics is that we don't have the luxury of the cold start that LLMs had, uh, with the internet. Fundamentally, a lot of this robotics data is three things: it's pixels, it's actions, and it's logic. And so we don't have access to a large amount of this curated and processed and clean data to be able to then train these models to understand what it is to be in the physical world. Uh, so a lot of our job is pretty much to, one, uh, create the different types of hardware such as sensorizing and wearables for humans, the different operations to go and collect and partner with individual contributors, businesses, tasks across different environments, and then ultimately post-process and own the quality gates to make these datasets useful and teach, uh, robots about the world.

   2. SHSonya Huang2:43

      Without having been given

3. 2:45 – 4:15

   Is robot data economically feasible without the internet?

   1. SHSonya Huang2:45

      the benefits of a giant, free, many trillions of tokens, maybe lots of trillions, trillions of trillions of tokens o- of, um, internet corpus, do you think that robotics, you know, having to pay for and operationally collect all the data that's going into the mix, is it even a feasible challenge? Like, is it economically feasible, uh, without having had the gift of the internet?

   2. JGJosh Gao3:08

      I think we, we still have the luxury of a, a lot of things that, uh, was given from the internet, so we aren't starting from zero ultimately. Um, I think you can see it across LLMs and data across LLMs. Pre-training data was always driven by the labs largely across web crawlers, things like that. It didn't become a, a large market. Today in robotics, we do see pre-training as quite a large market, and, and that could be and taken in several forms. So that could be that we are teaching, uh, video models how to generate, uh, more physics-grounded realities of what the world looks like, and we get to leverage all the work that has gone into VLMs and then different types of world models before to make that possible. Um, is it possible and scalable to get to that volume of data? I think that ultimately comes down to what commercial use case can you unlock at which levels of scales of data. And our strong belief is along that journey, that we will be able to unlock a lot of the simple, uh, pick and place tasks along the beginning of what we're creating. And towards that long tail, that is

4. 4:15 – 5:27

   Teaching robots about carbon life

   1. JGJosh Gao4:15

      going to be many orders of magnitudes larger than it is today.

   2. SHSonya Huang4:19

      I did a little homework before this. Your website is very whimsical. Uh, it says that you are teaching robots about carbon life. What, what does that mean?

   3. JGJosh Gao4:27

      I, I actually made the, uh, the website-

   4. SHSonya Huang4:28

      I love it

   5. JGJosh Gao4:29

      ... like, [laughs] like the night before we did our, our, our first tweet announcement or something like that. There's a, an idea in science fiction that is when robotics and AI has ultimately fully generalized and solved all tasks that humans can do, the final frontier is gonna be organic. Final frontier is gonna be biological. And you see this in, like, Asimov, you see this in Ender's Game, you could e- even in Hitchhiker's at the very end, the planet ultimately was this giant organic computer trying to process and understand what the meaning of life was or what the question was. Um, and so I think in the same vein for us, that is the human brain runs on just 20 watts. Uh, we are, we are self-healing. And so along the journey of teaching, uh, and getting robots to fully generalize, we'll, we'll, we'll slowly show them what, what, what carbon life is made of.

   6. SHSonya Huang5:23

      Hmm.

   7. JGJosh Gao5:23

      Yeah.

   8. SHSonya Huang5:23

      Love that. Okay, you come from an untraditional background

5. 5:27 – 7:03

   Cloud kitchens, stablecoins and the path to robotics

   1. SHSonya Huang5:27

      for robotics. I think you started in the fintech world and the sneaker world, if I'm not mistaken. Tell me about how you and your co-founder found your way to robotics.

   2. JGJosh Gao5:37

      It w- it was quite an interesting journey, yeah. For-- So before this, um, I was building essentially a, a Toast for Cloud kitchens. Um, this meant we did everything from back-of-the-house, uh, order management. Uh, at one point in, in my fourth year of university, I was skipping class to go, uh, deploy and install kiosks into rural Rochester, New York, uh, where we had to stay all nights replacing a lot of these, uh, different front of the houses for restaurants. Uh, my co-founder built a stablecoin. Uh, they ultimately sold to Coinbase, but, uh, he's been this very obsessive personality. He was, uh, top 1,000 in four competitive games, including League of Legends. And I think a big part of what brought us together, uh, one, we're both Canadian, uh, and two is we have this big, uh, science fiction grounding. And so the belief ultimately was that payment was a hard and difficult problem, but whatever we worked on next, we wanted to be something that seemed exciting, that the future was gonna be incredible, and robotics seemed like a great space to get into.

   3. SHSonya Huang6:40

      I remember walking into your office and then immediately seeing those, like, mechatronic robots and then the, the Kawasaki Corleo's on the wall.

   4. JGJosh Gao6:47

      [chuckles]

   5. SHSonya Huang6:47

      Like, the inner child in you must be very happy about what you're doing these days.

   6. JGJosh Gao6:51

      Yeah, absolutely.

   7. SHSonya Huang6:52

      Coming from a non-robotics background, what do you think your advantage- advantages are in building this business, um, and where has it presented challenges?

   8. JGJosh Gao7:01

      I think the advantages

6. 7:03 – 8:45

   Entering robotics with a fresh lens

   1. JGJosh Gao7:03

      is when we entered the space, we had to go and do this full discovery process. Um, we reached out to hundreds of folks that were in the space that we appreciated. We read every paper that we get our hands on. We went to visit the different robot labs. And what we saw in, in, in late '24, roughly, GPT was taking off, robotics was still evolving from this traditional, uh, robotics era, and it seems obvious to us at the very least that data and compute would become very important across the industry. And so moving towards that, we got to, uh, read and see some of the papers like Ego4D from a fresh lens and say, "Hey, there's about 4,000 hours of this human, uh, demonstration data." And when you actually went in to look at the data, you would realize some of the data sets would be daily life of folks like, uh, bowling, for example. There was very broad use cases and potential applications at the 4,000-hour scale. And so coming into, into that with, like, a, a fresh lens, we were able to, from a first principles approach, I'd say, uh, have a general intuition of what would be important and then iterate very closely with people that were much smarter than us that had, uh, that had different views of what the future would be and be able to stay close to iterate on what the product would become.

   2. SHSonya Huang8:22

      Yeah. Interesting.

   3. JGJosh Gao8:24

      Yeah.

   4. SHSonya Huang8:24

      It seems like robotics has just changed so quickly as a field that almost coming from the field can be an advantage and a disadvantage because people come from different eras of, you know, what's important in the training mix and things like that.

   5. JGJosh Gao8:34

      Yeah.

   6. SHSonya Huang8:34

      Um, maybe for our audience, just say a word on, like, what is egocentric data? And I think you really started your company kind of making a bet behind it. Obviously, it's much more distributed now,

7. 8:45 – 10:09

   What egocentric data is and why Mecka bet on it

   1. SHSonya Huang8:45

      but what is egocentric data? Why'd you decide to bet the company on it at the beginning?

   2. JGJosh Gao8:49

      Egocentric data is pretty much, uh, video data from humans demonstrating and doing tasks from a first-person POV. The high-level idea was that from the gap of human data to robotics data, you could close a lot of the vision gap. Uh, where robotics experience the world from POV, can we also show it what it is like to interact with the world from first person as well? Why we bet on, on it was, was pretty much when we explored the space. A lot of what we saw was robotic state at the time was teleoperation, which was humans controlling robots to do demonstrations. It was very hard to scale at the time. And from what we'd saw across GPT and a lot of the, the principles of scaling up, um, we had to find ways to scale up data much more significantly. And I think that some of our early intuitions were that humans and robots share a lot of similarities. Humanoids were just starting to take off. Humans and humanoids look the same. Why couldn't we turn humans into this fleet style learning that Tesla has to be able to all contribute into understanding and ground the physics of the world?

   3. SHSonya Huang10:00

      Hmm. And so what does that look like? You're, like, mounting a iPhone on somebody's forehead and just letting them go about their day? Or what, what does that look like in practice?

   4. JGJosh Gao10:08

      Yeah. When

8. 10:09 – 10:42

   The first pilot: aunties and uncles with a mobile app

   1. JGJosh Gao10:09

      we first started, our very earliest, uh, pilot was we created a mobile app. We got all the Chinese aunties and uncles to record themselves cooking, cleaning, doing tasks. That obviously evolved a, a lot. We, we, we tried a number of things. We did Umi data. We did all, all the sorts at the beginning. Uh, and that's evolved quite a bit into all the custom hardware that we have to produce now, um, and all the different partnerships that we have to have across different industries. And ultimately, we're sensorizing humans, and that's the broader category here.

   2. SHSonya Huang10:37

      Mm. How important is it to get force data from the sensors? 'Cause, you know, one of the things I've

9. 10:42 – 11:55

   Force data and tactile gloves

   1. SHSonya Huang10:42

      always thought about is in, in self-driving, you don't really need force data as much, right? Whereas with robotics, especially with manipulation, you really care about how hard am I squeezing that door handle or, or picking up that egg. And it seems to me that, like, video doesn't quite capture that. You might be able to approximate it, but doesn't really capture it. How important is capturing force data, and how do you think about getting that?

   2. JGJosh Gao11:01

      I think some of the substitutes for force data that we've seen would be approaches like risk cameras currently across both humans and robots, and being able to see the contact time between, uh, the hand and the different objects that you interact with. That obviously is kind of a, a hacky way ultimately to get to understanding how much force in newtons it takes to pull open a, a, a chair or, or twist a, a bottle cap. Um, I think force data will be incredibly important. It's actually one of the, uh, products that we've been building out in that when you iterate with many of dif- different, uh, robotics foundation labs, um, you realize to close this gap, one, we have to create new hardware. So we've been developing things like gloves for humans to wear, and in those gloves, being able to collect the information in how much force it takes to do your daily life and different tasks.

   3. SHSonya Huang11:53

      I'd love to just spend,

10. 11:55 – 14:41

    Inside the operations of a data factory

    1. SHSonya Huang11:55

       spend a bit of time on the actual operations of a data factory because I think it is-- There's so much money that's gone into the LLM data factories. There's so much money that is starting to be poured into and I think will be poured into the robot data factories, and the way they work is just, you know, mysterious and intriguing to me, so I'd love to ask you a bit about the operations of it, if that's okay.

    2. JGJosh Gao12:14

       Yeah, let's do it.

    3. SHSonya Huang12:15

       Okay. So how does it work? Like if I am, you know, if I wanna make an extra buck, do I just sign up on your website to start, you know, to put a camera on my, on my forehead and start sending you data, or how, how does it work?

    4. JGJosh Gao12:25

       So we, we actually have a lot of different approaches on the operator side. One would be we have a direct consumer approach. And so through direct mobile apps can you sign up from anywhere in the world, can you start contributing, can you, uh, can you register which profession you're in, and can you get streamed a sequence of tasks, uh, to do such as cleaning your bedroom, cooking your breakfast, cleaning up afterwards? Um, this is a very lightweight approach to get that diversity ultimately of this pre-training data. The second approach that you have to get into is ultimately there is so much more beyond the home to collect. And so we had to build out a lot of this B2B engine and this muscle of being able to recruit businesses across this incredible taxonomy of all physical human work, which is across bakeries, it could be a, a PCB manufacturing line, everything to-

    5. SHSonya Huang13:19

       Podcasting.

    6. JGJosh Gao13:20

       Podcasting. [both laughing] There's, there's not too much handwork in podcasting just yet.

    7. SHSonya Huang13:23

       Trying to make a quick buck, buck here. [both laughing]

    8. JGJosh Gao13:26

       We should have equipped ourselves. [both laughing] Um, and, and that's ultimately how contributors come into the mix. But the actual data engine for the data factory is so much more sophisticated and much more painful than folks can expect, and that is that a lot of the, the things that we have to distribute into the world didn't exist until we designed and produced this hardware. Even across tactile gloves is a fantastic example. There is nothing off the shelf that you can purchase and that you can wear and that reliably tells you how much force you are applying day to day and turn that into ready datasets for these different robotics foundation models. So we have to have that hardware iteration loop to have these three, four different product cycles, that engineering muscle to actually think and iterate what is going to be useful and then scale that production into the world. And that becomes this logistics of distributing into these thousands of homes, thousands of businesses to collect that, to upload that. We've had to, uh, set up these 100-gigabyte upload lines in a lot of the places that we operate in just to sustain the hundreds of petabytes of data that we have to deal with at this point. And then you have to process, you have

11. 14:41 – 15:42

    Heat-mapping the taxonomy of human work

    1. JGJosh Gao14:41

       to make sure your quality gates are incredible, and then all of the computer vision to make it useful ultimately.

    2. SHSonya Huang14:47

       So let's say you decide, okay, in my training mix, there's not enough podcasters, there's not enough, you know, I don't know, donut bakers. Do you go, do you go out and literally go knock on doors of donut bakers then to sign them up to the platform, or how does that work?

    3. JGJosh Gao15:00

       We'll, we'll generally track kind of like a heat map across different jobs, industries, and ways we think about diversity. And across that heat map, if ultimately we think, oh, not enough Sonyas podcasting, and we have to go get that-

    4. SHSonya Huang15:14

       [laughing]

    5. JGJosh Gao15:15

       ... that's actually why we're here today. This is our, this is our secret method. [both laughing] That's why there's cameras everywhere. [both laughing] Um-

    6. SHSonya Huang15:22

       No, you, n- you're never automating me out of this. [laughs] Um, no, I'm kidding. Please do. Uh, okay, interesting. Like, I've heard that, you know, the, there's a lot of quality issues in robot data collection generally, and that a lot of the hours of robot data sample that get sent in end up not being accepted. Um, why is that,

12. 15:42 – 17:44

    Quality gates, fraud and the McDonald’s POV videos

    1. SHSonya Huang15:42

       and what are you doing about that problem?

    2. JGJosh Gao15:44

       This lesson came from even originally the different types of robot data. There are, have been these very large scale robot factories that have spun up from just large scale warehouses repetitively doing tasks trying to scale up teleoperation data. Uh, some of the learnings there came from the research in that there is some threshold of diversity where you oversaturate how much robot data you need or how much data in general you need. Across human data, quality, you can imagine, is actually quite a bit of a moving target as we grow. The amount of edge cases that you have in the real world that you have to deal with are very extreme, and you will always see something new that you haven't seen. So for us, it's been about can you think about the heuristics to be able to catch a lot of the different things as it's processed? Can you have the different engines to escalate very quickly? And ultimately, we will learn new things about quality and diversity that weren't true a week ago even. And so the nimbleness to move very quickly to create and adapt your engine to catch that, uh, and make sure it-

    3. SHSonya Huang16:50

       What's an example of something that you would catch?

    4. JGJosh Gao16:52

       Well, uh, I, I think, I think fraud is an, is an underestimated one across data industries in general. We're ultimately creating data that doesn't exist yet versus when you're filtering through the internet. And so when you are, you're, you're coming across very creative and, uh, interest-

    5. SHSonya Huang17:08

       Like people are sending generated videos or what? [chuckles]

    6. JGJosh Gao17:10

       Yeah, very creative people where, uh-

    7. SHSonya Huang17:12

       Huh

    8. JGJosh Gao17:12

       ... I think at one point we have folks, uh, recording YouTube videos like POV working at McDonald's and trying to submit that as, as contribution pretty much. [both laughing] Um-

    9. SHSonya Huang17:24

       Wow, never underestimate how entrepreneurial people can be. That's amazing.

    10. JGJosh Gao17:29

        Yeah, very much. Uh, and, and so the- these have been like part of the painful lessons where, yes, you have to create a lot of the gates, but ultimately you have to also be there where collection is happening. You have to go see how it's done. You have to figure out an incentive model

13. 17:44 – 19:33

    Earning a margin as a data intermediary

    1. JGJosh Gao17:44

       that actually, uh, incentivizes people to create good contribution.

    2. SHSonya Huang17:48

       How do you think about your right to earn a margin long term? And, you know, I can imagine a Mecka competitor comes to Sonya and says, "Sonya, I really want your podcast data." Um, I could imagine that, you know, the margin that you're able to own as an intermediary ends up getting squeezed.

    3. JGJosh Gao18:05

       I think the contributor, uh, amount in terms of the, the margin cost is actually not a large-- not the largest factor at play here in terms of the end-to-end making sure this data is useful. So even though that baseline is always gonna be there and maybe it becomes more and more uncompetitive, um, there will be ways to differentiate and make sure that your data is verifiably high quality at very large scale with the right hardware, the operations, and the software. I think when we think about these products, ultimately it is that we do want to scale up and mature these products. And just like any, any product that, uh, begins to scale up, you want to optimize it. But over time, you may have different products that evolve. And so when you think about this egocentric data of podcasting as simply one of your SKUs, and you then realize that there is going to be different types of data that are gonna be important, such as maybe we move from the, the Chinese restaurant, uh, from the side of the road, can we start moving into the Michelin star restaurants over time? Can you get more and more closer into what it means to be Federer than it means to be an average tennis player? So these products will change, uh, and, and, and how these products are created will also change.

    4. SHSonya Huang19:23

       Really interesting. What scale of data has already been collected, do you think? And how much data do you think is gonna be collected over the next few years?

    5. JGJosh Gao19:31

       I think we finally entered

14. 19:33 – 20:55

    Millions of hours vs. Tesla’s decade

    1. JGJosh Gao19:33

       a world where we are in the millions of hours of data. And coming from the world where Ego 4D of 4,000 hours two years ago, where we were then, uh, is a huge step up. But then comparatively, and you think about autonomous driving as an example, uh, Tesla spent over a decade solving for two degrees of freedom ultimately, of acceleration and direction, millions of cars on the road. They have a lot more than a few million hours, uh, of video data at this point. So relatively, we're still many orders of magnitudes away from where we have to be ultimately.

    2. SHSonya Huang20:14

       So on one hand, that sounds like opportunity for you. That sounds like a lot of revenue for the robot data factories. On the other hand, as a, you know, as somebody that, you know, invests in the robot foundation models, I look at that and I say like, "Oh, shoot, are we ever gonna get to the promised land?" 'Cause I, I've, I've done the same math as you and like a car has two or three degrees of fre-freedom, a robot has-

    3. JGJosh Gao20:33

       Even dozens in the hand.

    4. SHSonya Huang20:35

       Yeah, exactly.

    5. JGJosh Gao20:36

       Mm-hmm.

    6. SHSonya Huang20:36

       And so you combine that with like you probably need force data and there's a way more... You know, it's just a way harder environment to operate in, and I get to like you might need billions, trillions, quadrillions of hours. Is there even enough GDP in the world to support the

15. 20:55 – 22:02

    Why it isn’t binary: unlocking commercial tasks along the way

    1. SHSonya Huang20:55

       cost that to bring that in? Like, what do you think?

    2. JGJosh Gao20:57

       I think it's about unlocking commercial use cases as we go. So it doesn't have to be so much binary in that, oh, one day we've unlocked robotics or not. It is going to be a slow creep up into the world. Um, a great ways to get there is that with some of this base pyramid built up, where we are seeing the generalizations, we are seeing these scaling laws at play, that gives people the courage to scale up. And with that courage, can we, one, have some generalizability in some of these environments? Can we then also collect a lot of the in-domain, in-task, uh, data that closes that gap in post-training? So maybe I can collect data exactly from this room, exactly in this podcast demonstration, uh, to actually close the gap of what is, in a lot of cases, four nines of reliability in these, in these real world environments.

    3. SHSonya Huang21:49

       That's a helpful framework. So it's not about a binary of getting from zero to one. It's about we're gonna increasingly get, you know, robots that can do an increasing set of capabilities.

    4. JGJosh Gao22:00

       And we will unlock a lot of tasks along the

16. 22:02 – 23:24

    Home data is plentiful; home robots are the hardest

    1. JGJosh Gao22:02

       way.

    2. SHSonya Huang22:02

       Yeah. One curiosity, though, is that, you know, the data that is most plentiful right now is like, it seems like home data, right? Just 'cause like everyone, everyone can film that. It's pretty easy to get. But the robot in the home almost seems like one of the most difficult environments to deploy in. And so I'm curious what applications you think will hit first in robotics, 'cause like where the data's most readily available seems almost the opposite of where robotics might be easiest to deploy.

    3. JGJosh Gao22:29

       The, the home is gonna be some of the environments that we have to be most diligent on by far. And a lot of these benchmarking and evals that have just started to mature across robotics, making sure that the, the robot doesn't fall on the baby in the homes, for example. Um, I think the initial tasks will be much more constrained, uh, as we go along that spectrum in that very goal-oriented, where you can hill climb some of these tasks, where you have existing SOPs for what it means to do that job and that work. Um-

    4. SHSonya Huang23:02

       Like a laundry folding robot.

    5. JGJosh Gao23:04

       It could be laundry folding. Laundry folding, uh, we, we, we spent, uh, about a year and a half plus trying to solve, and I think it will absolutely get solved in the near term. A lot of these pick and place, these packaging tasks that you see on assembling lines that had to be manipulation bots that couldn't be solved by traditional robotics, uh, we will get there within the next year or two.

    6. SHSonya Huang23:22

       Hmm.

    7. JGJosh Gao23:23

       Mm-hmm.

    8. SHSonya Huang23:23

       It is

17. 23:24 – 24:35

    Useful robots can be deployed today

    1. SHSonya Huang23:24

       a good transition to talking about deployment. I'd love to talk to you about, you know, robot deploy- deployment overall as a concept. Um, you've said that useful robots can already be deployed today. Where do you think that's the case?

    2. JGJosh Gao23:36

       I think people underestimate how diverse robotics can be. Uh, and if we, if we constrain ourselves to, to one, humanoids are incr-incredibly popular, it will be, uh, a very popular terminal state of robotics as well. But beyond that, there are so many different form factors that exist in robotics today. You could look at different markets, even across, uh, hotels in China or Southeast Asia, for example, where there are robots that are doing services, that are doing cleaning, that are delivering from after your, your Uber Eats gets to the door, from the front desk to your hotel room door that exists. Many of these bots are widely deployed across the world today. They are just starting to get popularized across the US and North America. I think some of the first bots that have to get deployed will be even just the industrial cleaning bots, for example, uh, a lot of, like, thematic robotics that are out there. And then slowly stepping

18. 24:35 – 26:42

    When robotics gets an FSD-style flywheel

    1. JGJosh Gao24:35

       up and up from there, where you start to solve early manipulation tasks along that spectrum, all the way up until we get to that humanoid solving out of the box any task in the future.

    2. SHSonya Huang24:46

       You mentioned Tesla and FSD earlier in this show. Um, when do you think we're gonna get to the point where there is a little bit of an FSD-like flywheel? And when we get to that point, how do you see your role as a company evolving?

    3. JGJosh Gao25:00

       When we will get there, um, I think we're a whiles away, but it'll be a slow transition in that when you are slowly solving some of the commercial tasks, you will also be able to finally get some of these flywheels. Then you can get some of the in-deployment data where you get the failure cases, you get the correction data. Um, that will be absolutely important in starting up this robot flywheel. How far that is depends on that spectrum and across saturating different types of tasks and domains that aren't completed or solved for yet. So it's quite a whiles away. When we do get there, the thing is, I, I, I believe that Mecka will look very different along that process, in that, one, w- we're not fully subscribed to only egocentric data. We believe generally that a lot of different SKUs, the entire data soup is important. And so will we be producing the Michelin star data at, at some point in the future? Will we be teaching some sort of edge case about the real world that the robots haven't gotten to yet? If we believe that the world is incredibly vast and rich in detail, um, there's gonna be no shortage of things to teach. And then even in that world, we do have this angle where we believe that Mecka will very much contribute to the deployment of robots. And so in, in some of these environments, in closing that gap of being able to do that task in that domain, in that environment, can we actually be the platform where we collect that domain data, we post-train, we help support the enterprises and white glove hold them into what it means to be in a world filled with robotics? We see Mecka maturing into that.

19. 26:42 – 28:21

    Deployment diffusion and the modern systems integrator

    1. SHSonya Huang26:42

       I'd love to get your predictions on how you think deployment diffusion is gonna work out in the physical world. Um, and maybe if I can analogize what's happened in the LLM world, there's, like, a few different ways that deploy codes have happened, right? There's vertical specific application companies. Uh, there's the foundation model companies themselves that are scaling up go-to-market. And then there's, you know, all these FD deploy co, service co, whatever you wanna call them, you know, just armies of people, um, helping enterprises adopt AI at scale. Those are-- seem like the broad vectors that are disseminating tokens across the economy. What do you think the analog will be in the physical world?

    2. JGJosh Gao27:22

       We, we believe very much in both the vertical and the horizontal across robotics, so the full verticals, um, that are very task-specific, all the way from hardware to models to data they're tackling. We also believe, uh, believe in an ecosystem where there's gonna be very vast and different shapes and sizes of hardware of robotics, uh, and there will be different types of open models that exist in the world. I think it will look something like a modern robotics version of a systems integrator, that in a traditional version, you are installing the product, it's incredibly competitive, it is tight margins. There is now a world where you can actually install and help make that product become more data-driven and productive over time. So can you actually, uh, have the bot you installed last week do something that it didn't think was possible, uh, a few, a few days ago even? Can it help you start expanding the scope of what your business is capable of doing? Uh, that's ultimately the milestone that we're, we're hoping

20. 28:21 – 30:08

    Favorite vertical robot categories; mining an asteroid

    1. JGJosh Gao28:21

       to, to aim for.

    2. SHSonya Huang28:22

       Okay. But you would tell me as an investor, now give me some investing advice. You, you'd invest in both the general and the vertical robots then?

    3. JGJosh Gao28:28

       Yeah.

    4. SHSonya Huang28:28

       What are your favorite vertical robot categories?

    5. JGJosh Gao28:31

       I think the best ones will be the industrial bots to start. Those are, are, are, are, are gonna be ones that already have a lot of traditional robotics in the mix that are task constrained like we talked about.

    6. SHSonya Huang28:42

       Like data center maintenance robots?

    7. JGJosh Gao28:44

       Yeah, you just don't, don't have to expect a baby to appear, uh, and have to think about that a- and not fall on it, for example. So the, the world is much more constrained than in the wild. Um, other categories we're excited about are, are, are just spaces that are even feasible and possible, uh, early on. And so some of the approaches in the hotels, for example, where you could actually start delivering from the front desk to the hotel rooms, um, be able to actually have some manipulation to hand off, um, are exciting to us, and then places that just humans were not capable of. A, a joke that we make in, in some of our all-hands in the company would be if one day Mecka was able to help deploy a robot that mines an asteroid, we've probably done a pretty good job. [laughs]

    8. SHSonya Huang29:33

       [laughs]

    9. JGJosh Gao29:34

       It's just a, it, it's just, it's a job and a task that completely doesn't exist, that humans can't do, that we have no idea how to do just yet, but it seems inevitable.

    10. SHSonya Huang29:42

        Yeah. I love that. Reminds me of, you know, even... I'm not thinking interplanetary yet, but, you know, like, have you heard of Gecko Robotics?

    11. JGJosh Gao29:50

        Mm-hmm.

    12. SHSonya Huang29:50

        Um, what they do to help maintain ships and things like that.

    13. JGJosh Gao29:53

        Yeah.

    14. SHSonya Huang29:53

        I think it's really cool.

    15. JGJosh Gao29:54

        Absolutely.

    16. SHSonya Huang29:55

        Can you say a word on, and whatever you're willing to share publicly, the slate of hardware that you've developed and where, where the labs currently are most interested in, in collecting bespoke data?

    17. JGJosh Gao30:06

        Some of the, uh,

21. 30:08 – 30:55

    The hardware slate: Waymo-fying people

    1. JGJosh Gao30:08

       iteration came from when we started with the mobile app, we quickly were able to scale that. There were a lot of virtues in that product. Um, the downsides ultimately were you don't get as much rich information about the world as you could have. And so some of the things that we had to develop very early on, uh, were things like Waymo-fying people ultimately. And in this sensorization, can you build things like these baseball caps that are equipped with different cameras and sensors? Um, we build a central hub that then connects to all these different accessories, such as the tactile gloves, external cameras, and more, all of which is trying to capture how do you interact with the world on a day-to-day, and hopefully be able to clone that behavior.

    2. SHSonya Huang30:53

       Hmm. Interesting. What is

22. 30:55 – 32:04

    EgoVerse

    1. SHSonya Huang30:55

       EgoVerse?

    2. JGJosh Gao30:56

       EgoVerse, uh, so Professor Dan Feiz, Ryan Zimmer, these are some of the earliest cohorts of researchers that believed in egocentric data. Uh, we had the privilege to be, uh, an early contributor in that consortium ultimately of different folks that think scaling up human data would be important for robotics. And so, one, we created a, and contributed a large amount of different types of egocentric data across different businesses, industries, home environments. Uh, we shared all of the, uh, action trajectory data, so all of the post-processing, the computer vision, and the quality gates. Uh, and we contribute all the, uh, platform and web tooling, pretty much. So the data loading, the infrastructure. And so it is ultimately gonna be, uh, to, to repeat their vision on this, is ideally we can bring together a lot of the robotics industry because hopefully we can lift up everybody in this wave and share the tooling, the data, and the lessons just because we are so far behind as an industry to get there.

    3. SHSonya Huang32:02

       Is it fair to say that the robotics industry

23. 32:04 – 33:54

    The data pot of soup: the converged training mix

    1. SHSonya Huang32:04

       has kind of converged on a training data mix that is, you know, internet-scale data from YouTube and the like, plus a bunch of egocentric data, plus tele- uh, UMI, and then plus teleop, and like maybe there was like a triangle of like amount of data on each. I think Jim Fan had like a really nice visual on this, um, at a Sequoia event. Is that kinda like the mix that you see? And then amongst your customers, has everybody kind of converged on a very, very similar mix?

    2. JGJosh Gao32:32

       It, it's, it's fun to see that even a year ago, uh, many folks would argue and say, "Oh, no, it's only UMI data that's gonna work. It's only egocentric data that's gonna work." And you slowly realize, to your point, all of this is essentially a data pot of soup. And in that pot of soup, the, all the ingredients are going to im- be important at some point. I think what we're seeing is a convergence in cross, uh, what that recipe might be today, and that does look like some version of a million hours of human data across egocentric and different, different types of modalities. Uh, some tens of thousands of hours of UMI data, and then some last mile, uh, teleoperation data and in-domain, in-distribution data in a, in just a couple hours even of that demonstration. I think we are seeing that. But over time, and why, why we don't, uh, subscribe aggressively to even one religion here is that things like tactile data haven't even entered the mix yet. And if there is this belief that these, these models ultimately will need to learn how much force it takes to apply, uh, towards the door handle, for example, these will have to come into the data pot of soup. Simulation will come into da- data, data pot of soup. And they all, all will be important to converge on.

    3. SHSonya Huang33:53

       One of the,

24. 33:54 – 36:02

    Move 37 for robotics: AlphaGo and Federer

    1. SHSonya Huang33:54

       I guess, objections that I hear most frequently to, um, to the types of data being collected are, you know, you could watch millions of hours of tennis data, and you still wouldn't know how to beat Roger Federer, right? And I'm curious what you think of that and what it means for where we might be headed in terms of the way robots learn.

    2. JGJosh Gao34:14

       I, I, I love the analogy of AlphaGo ultimately. When you talk about, uh, robots playing tennis, you gotta talk about robots playing, uh, Go. And I think with Lee Sedol and, and, and all the work that DeepMind did, at the very beginning, it was large-scale behavior cloning of different experts playing Go. Then the next step was the self-play version of with the rules of the game, can you ultimately simulate all the different ways until you come up with the magic of Move 37 ultimately? Um, which was so out of scope of what humans could have thought in that moment on the Go. For us though, in our case of robotics, for, for Go, the rules of the game are the, i- i- is the entire, uh, circumference of the world that you will operate in. In robotics, you have the incredible rich detail that the, that the physical world has, and we ultimately don't have a world simulator that can recreate that today. And so to close that gap, a lot of the egocentric data today is to, one, help us ground the basic physics, uh, and the basic, uh, behavior cloning that you have to get to. It is also contributing to creating what this world simulator might look like. And then somewhere along the way, I think we will be able to create and have robots playing tennis like Roger Federer.

    3. SHSonya Huang35:39

       Hmm. Do you think they'll mostly be doing self-play in simulation or in the real world?

    4. JGJosh Gao35:46

       I could see, I could see a hybrid ultimately. I think if we can get simulation, and we don't have to fully reconstruct the world to, uh, to its detail like right away, but I, I see self-play both in the physical and in simulation.

    5. SHSonya Huang36:00

       You used the word world simulator

25. 36:02 – 36:58

    World models and the training mix

    1. SHSonya Huang36:02

       a couple times there. What are your thoughts on, I guess, what's happening with world models, which I guess are being used somewhat interchangeably with video models as a term right now, um, but world models and like their contribution to the robot data training mix?

    2. JGJosh Gao36:17

       I, I, I think the potential and possibility to have what we just talked about in, in terms of self-play, to even just dream what it means to drop a cup and have it shatter without actually doing it ... is incredibly valuable for what, what could happen. I see the value in that we get to leverage a lot of the work from the internet and YouTube, for example, in this pixel prediction ultimately. And if we can translate that back into what these action trajectories would be, um, it could help us accelerate quite a bit.

    3. SHSonya Huang36:50

       Yeah. Yeah. Interesting. So you mentioned that we've kinda converged on this, like, pyramid of data max. Do you think this pyramid will, like, scale

26. 36:58 – 39:05

    Will scaling the current mix get us to the promised land?

    1. SHSonya Huang36:58

       if we just scale it up? Do you think that takes us to the promised land? And to me, promised land is people like saying GPT-3 moment for robotics. I no longer like that. I just like, when will a robot actually fold my clothes or cook a meal for me? Um, do you think the current data mix will get us to that point, or do you think we're, like, new breakthroughs and algos and data are gonna be needed to get there?

    2. JGJosh Gao37:19

       I, I think if we scale it up some order of magnitude in that ratio, uh, we will solve a lot of tasks along the way. So some of those tasks could be a lot more constrained environments. They could be within business operation tasks, very specific SOPs you give. Um, in that ideal world where we have a drop-in robot and it solves any task by just looking at it, can play tennis just like Roger Federer out of the box, um, in all of the i- uh, crazy complexities of the actual world scenarios, like the weather, for example. I think it will be some way, and I don't think this is what it will look like ultimately. Um-

    3. SHSonya Huang38:02

       What will it look like?

    4. JGJosh Gao38:04

       Well, we still have so much to add into the mix still, and I think, I think that is gonna be how can we start introducing other modalities into the mix? This is gonna be that ta- tactile data. I, I like the approach a lot of the models converging towards an omni model approach.

    5. SHSonya Huang38:22

       Mm-hmm.

    6. JGJosh Gao38:22

       And where you can feed it any type of input, that is the observation, whether it is force, pixel, even smell in a very extreme case, and then have some sort of output, uh, in, in whether it's action, voice, or, or, or different type of, types of controls.

    7. SHSonya Huang38:43

       Are there even chips that can sense smell?

    8. JGJosh Gao38:45

       I... It's, it's, it's pretty, it's pretty early. This olfactor, uh, approach is, is very early.

    9. SHSonya Huang38:51

       Fascinating.

    10. JGJosh Gao38:52

        But I, I like to imagine that there's gonna be an approach in the world where, where that gets unlocked.

    11. SHSonya Huang38:55

        Yeah. The results of, you know, GPT-6 Astra is generalizing incredibly well to robot tasks. I'm curious what you make

27. 39:05 – 40:50

    Will the AGI labs steamroll robotics?

    1. SHSonya Huang39:05

       of that and what it means for all these pure play robotics companies. Like, are they ultimately just gonna get clobbered by the AGI labs and, like, the bitter lesson is just gonna steamroll robotics as well? What do you think?

    2. JGJosh Gao39:17

       It takes a lot of courage to scale up data with imperfect information, and I think a lot of these labs have the DNA in themselves to take on that courage, uh, early on. I think what we're, we're seeing and what, what's exciting is that, um, with the scale-up of robotics data and even just adding into, uh, a lot of the pre-training mixes of these main LLMs and th- these foundation models, uh, it really is that you can start to generalize in a lot of different domains. I, I'm excited about the omni model approaches where you can introduce a lot of different types of observation, whether that's force or whether that's action trajectories, uh, into the mix and allow it to generalize. Um, I think there are also gonna be a lot of arguments where you want to be much more vertical than a fully generalized model that can do anything. So s- uh, somewhere along that time horizon, that model that can do everything is gonna be very good at a lot of things, but some of these specialized models, some of these more targeted robotics foundation models might become very excellent into four nines of capability and reliability in some of these tasks.

    3. SHSonya Huang40:32

       Hmm. Okay. What I heard reading between the lines just now was that the, you know, the fact that the latest foundation models are great at robotics is not actually that surprising, um, in part because there actually is quite a bit of robot data that's been now thrown into the training mix. Is that fair?

    4. JGJosh Gao40:48

       It's actually quite expected when you think about it.

28. 40:50 – 42:44

    Sci-fi roots and the Kawasaki Corleo

    1. SHSonya Huang40:50

       Very interesting. [lip smacks] Um, I'd love to close by going back to your Sci-Fi roots. Um, you started this company because, you know, you could imagine a version of the world with robots everywhere. Paint a picture of what, what you think the world looks like.

    2. JGJosh Gao41:06

       I, I think this was, uh, the, the funniest moment was probably when you visited our New York office, and in our main conference room, we have hung up on the wall actually the, uh, the Kawasaki Corleo.

    3. SHSonya Huang41:17

       I love that.

    4. JGJosh Gao41:17

       [chuckles] Which is-

    5. SHSonya Huang41:19

       The robot panther dog horse, whatever it is.

    6. JGJosh Gao41:22

       The panther dog horse.

    7. SHSonya Huang41:23

       Okay. [chuckles]

    8. JGJosh Gao41:23

       Exactly. So, so it's pretty much a, a robot panther dog horse. [chuckles] Um, and it's the most ridiculous video and trailer you've ever seen, pretty much.

    9. SHSonya Huang41:32

       My favorite part is that you guys convinced, uh, Anas that it was actually you and your mom on it. [chuckles]

    10. JGJosh Gao41:38

        Yeah, yeah. Th- th- there's, there's a clip pretty much where this, uh, this fantastical, like, Asian lady and her, and her son are riding this green meadow over a, a, a little river. Uh, and the Corleo, this horse dog panther robot, is jumping over the river.

    11. SHSonya Huang41:54

        [chuckles]

    12. JGJosh Gao41:55

        And so we told Anas this, this was pretty much, uh, us as when we were kids, and we generated it.

    13. SHSonya Huang41:59

        [chuckles]

    14. JGJosh Gao42:00

        Um, but what is wonderful is how absurd it is. I, I very much love the philosophy of Hitchhiker's in just how absurd the world could be and how absurd the future could be. Um, and in this world that is full of robots, I would love to believe that there's gonna be such crazy entrepreneurs and, and, and people that believe that form factors like this would be interesting. Um, and I think that the, the, the shape and size of robotics will look something, uh, like a world where we come up with wacky, uh, robots that could do anything that you haven't even thought of yet.

    15. SHSonya Huang42:36

        I love that

    16. JGJosh Gao42:37

        ... like mining asteroids. [laughs]

    17. SHSonya Huang42:38

        [laughs] Um, order of magnitude, how many robots will there be in the world in a decade?

    18. JGJosh Gao42:42

        Many, many times

29. 42:44 – 45:11

    How many robots in a decade?

    1. JGJosh Gao42:44

       more than there are people.

    2. SHSonya Huang42:46

       Many orders of magnitude more?

    3. JGJosh Gao42:48

       Than there are people, yes.

    4. SHSonya Huang42:49

       Wow.

    5. JGJosh Gao42:50

       In, in, in, in dec- in a decade plus. Maybe a-

    6. SHSonya Huang42:52

       Like 100 billion robots?

    7. JGJosh Gao42:54

       Yeah.

    8. SHSonya Huang42:55

       That's really cool. 100 billion panther dog horses?

    9. JGJosh Gao42:58

       Panther dog horses, at least another order of magnitude-

    10. SHSonya Huang43:01

        [laughs]

    11. JGJosh Gao43:01

        ... more than that. [laughs]

    12. SHSonya Huang43:03

        Love it. Um, awesome, Josh. I think that what you're building is, like, operationally really impressive and very cool how you are scaling up the data collection efforts to enable kind of this robotics golden age and this sci-fi future that you've dreamed of since you were a kid. And so thank you for, for joining the pod today to tell us both how you're building the business and how you see the future. It was really fun.

    13. JGJosh Gao43:24

        Thank you so much for having us. Thank you, Sonya. So we actually brought you a present. [laughs]

    14. SHSonya Huang43:30

        [gasps] I love presents.

    15. JGJosh Gao43:31

        This is a, this is kind of like a full circle moment for us.

    16. SHSonya Huang43:34

        Is this what I think it is?

    17. JGJosh Gao43:36

        Who knows? Who knows?

    18. SHSonya Huang43:37

        [laughs]

    19. JGJosh Gao43:38

        Yeah. All right, you're gonna have to open it up on, on the podcast, so.

    20. SHSonya Huang43:40

        Okay, this is amazing, and look at the... Hold on. Can we... You know, I could be one of those YouTubers that does the unboxings.

    21. JGJosh Gao43:47

        [laughs]

    22. SHSonya Huang43:48

        Look, I love this. The Mecka stickers are A+. Okay, I think this is what I think it is. Let's see.

    23. JGJosh Gao43:53

        We're, we're gonna actually, actually come out with a, a, a series of Mecka figurines-

    24. SHSonya Huang43:58

        Oh, I love that

    25. JGJosh Gao43:58

        ... and stickers, because they're all, like, giant robots, human pilot stuffs. That'll be fun.

    26. SHSonya Huang44:04

        Okay. I'm so excited. Let's see if this is what I think it is.

    27. JGJosh Gao44:07

        [laughs]

    28. SHSonya Huang44:10

        Ta-da. [laughs] I love it.

    29. JGJosh Gao44:15

        I love it.

    30. SHSonya Huang44:15

        Oh my gosh. Okay. This is the-

Episode duration: 45:11