The GPT Moment for Robotics Is Here
Quan Vuong (guest), Jared Friedman (host), Jared Friedman (host), Garry Tan (host)
In this episode of Y Combinator, featuring Quan Vuong and Jared Friedman, The GPT Moment for Robotics Is Here explores foundation models and cloud control are accelerating robotics startup boom Robotics progress accelerated when vision-language models began transferring semantic knowledge into robot actions, as shown by projects like PaLM‑E and RT‑2.
Foundation models and cloud control are accelerating robotics startup boom
Robotics progress accelerated when vision-language models began transferring semantic knowledge into robot actions, as shown by projects like PaLM‑E and RT‑2.
Open-X demonstrated cross-embodiment training—pooling data from many robot types—can outperform single-robot specialists, suggesting emerging scaling laws in robotics.
The biggest constraint is not algorithms alone but data generation/capture and operational infrastructure for collecting, managing, and evaluating real-world robot data.
Mixed-autonomy deployments (robots operate with minimal human takeover) are now viable for real businesses, as shown in laundromat folding and warehouse packaging demos.
Cloud-hosted models can control robots in real time via API calls using action-chunking/pipelining, reducing on-robot compute requirements and enabling faster iteration as models evolve.
Key Takeaways
Robotics is becoming a foundation-model problem, not just a control-stack problem.
Language/vision-language models supply common-sense semantics and planning priors, and adapting them with robot data enables surprising generalization to unseen objects and instructions (e. ...
Get the full analysis with uListen
Cross-embodiment training can beat single-robot specialization.
Open‑X results described a generalist model trained on data from ~10 robot platforms performing ~50% better than embodiment-specific specialists, implying abstraction (“how to control robots”) emerges with diverse data.
Get the full analysis with uListen
The true bottleneck is operational: collecting, capturing, and evaluating robot data at scale.
Unlike the internet for text, robotics lacks a naturally aggregated dataset; progress depends on building repeatable pipelines for data ingestion, quality, labeling, and evaluation—especially as task duration and capability grow.
Get the full analysis with uListen
Mixed autonomy is the practical bridge to full autonomy and scalable economics.
Deployments can start when mistakes are tolerable and humans can intervene; with corrections and continuous exposure to edge cases, systems improve until they approach full autonomy and can reach economic break-even.
Get the full analysis with uListen
Real-world demos show robots are moving from lab tricks to operations-ready systems.
The Weave laundromat folding example emphasizes deformable-object generalization, while the Ultra warehouse pouching example emphasizes long-duration autonomy under changing lighting and real production constraints.
Get the full analysis with uListen
Cloud-hosted control is a major unlock for deploying large models on cheap robots.
By pipelining inference inside the control loop and using “action chunks”/real-time chunking, robots can query a remote model via API while maintaining smooth control—avoiding expensive, soon-obsolete onboard compute.
Get the full analysis with uListen
Starting a robotics company now looks like a workflow-and-ops playbook.
Quan’s recipe: deeply understand an existing workflow, pick high-leverage insertion points, use scrappy/cheaper hardware, build data+eval loops, run mixed autonomy to break even, then scale fleet size.
Get the full analysis with uListen
Notable Quotes
“Our mission is to build a model that can control any robot to do any task that it's physically capable of.”
— Quan Vuong
“If you simply take the data and absorb it into a model… you can compare it to the specialist… And… it was 50% better.”
— Quan Vuong
“In robotics… if you wanna add two years to your PhD, just work on a new robot platform.”
— Quan Vuong
“Folding laundry… has always been… the Turing test for robotics.”
— Jared Friedman
“Almost all of the robot evaluation that we run… the model [is] hosted in the cloud… querying an API endpoint… and getting back action that then execute directly on the robot.”
— Quan Vuong
Questions Answered in This Episode
What specific evidence makes you confident robotics has “scaling laws” now—beyond the Open‑X 50% result?
Robotics progress accelerated when vision-language models began transferring semantic knowledge into robot actions, as shown by projects like PaLM‑E and RT‑2.
Get the full analysis with uListen AI
Open‑X helped with data availability but not evaluation; what would an “ImageNet for robotics evaluation” look like in practice?
Open-X demonstrated cross-embodiment training—pooling data from many robot types—can outperform single-robot specialists, suggesting emerging scaling laws in robotics.
Get the full analysis with uListen AI
You distinguish data generation vs data capture—where do you think the biggest ‘hidden’ pools of uncaptured robot data exist today (warehouses, factories, hospitals, homes)?
The biggest constraint is not algorithms alone but data generation/capture and operational infrastructure for collecting, managing, and evaluating real-world robot data.
Get the full analysis with uListen AI
How do you manage safety and fault containment when the control model is in the cloud and latency/packet loss are inevitable?
Mixed-autonomy deployments (robots operate with minimal human takeover) are now viable for real businesses, as shown in laundromat folding and warehouse packaging demos.
Get the full analysis with uListen AI
In the Ultra warehouse deployment, what were the most common intervention reasons, and which ones look most solvable via more data vs model changes?
Cloud-hosted models can control robots in real time via API calls using action-chunking/pipelining, reducing on-robot compute requirements and enabling faster iteration as models evolve.
Get the full analysis with uListen AI
Transcript Preview
The equation, I think, for starting a robotic business has changed and will continue to change at an accelerating pace because the upfront cost is not that high anymore.
Everyone's sort of spending a lot of time in the digital world, and it feels like, you know, now is the time to start thinking about the world of atoms.
You literally just gave people the playbook for how to build a vertical robotics company.
This has really been our mission from the start, is to create that Cambrian explosion.
It still, like, blows my mind. I didn't know if this would exist even in my entire lifetime.
[upbeat music] Welcome back to another episode of The Lightcone. Today, we have a very special guest, Quan Vuong. He's one of the co-founders of Physical Intelligence, which we think might be the robotics AI lab that brings about the GPT-1 moment for all of robotics. Quan, thank you for joining us.
Pleasure to be here. Has been a long time admirer of YC, and our mission is to build a model that can control any robot to do any task that it's physically capable of, and to do so at such a high level of performance that's gonna be useful to people in all walks of life. And so GPT-1 for robotics, you know, what is it? You know, is the ChatGPT moment for robotic real? Our perspective here is that, um, we want to build a model that's really intelligent. We want to build a platform that allow us to externalize that intelligence to the rest of world and allow them to use it to build very interesting application in all sorts of vertical and robotics. And w-we think that it's gonna be more like a peeling an onions analogy, where you start from a really strong base model that have all sorts of common sense knowledge and already works to some extent on your robot. Um, you have then a mixed autonomy system, uh, very similar, for example, to a autonomous driving car today. Um, and then you actually deploy that system to do a real job. That system might make mistake. Um, it's okay. Um, and then over time, by actually exposing the system to the complexity and the edge case of the real world, that system get incrementally even just slightly better over time every day. Um, and you know, one day you wake up, and you suddenly have a system that is just fully autonomous and just provide tremendous value.
It might be helpful to give the audience a bit of a mini history lesson on why robotics is so hard, and there's been a lot of breakthroughs in the last two years. And I mean, just to simplify, the robotics problem is three pillars, semantics, which I think we got a lot of analogs in with language models that somehow we ported into robotics, then you have the planning, and then the last thing is control, which needs to be done in real time and interact with a environment that changes. Walk us through the seminal papers that a lot of the team of Pi Robotics publish that gave you the inkling that the GPT-1 moment is near, and that started in twenty twenty-four.
Install uListen to search the full transcript and get AI-powered insights
Get Full TranscriptGet more from every podcast
AI summaries, searchable transcripts, and fact-checking. Free forever.
Add to Chrome