a16zSolving the Hardest Problem in Robotics | Fei-Fei Li with a16z
CHAPTERS
- 0:00 – 1:08
Spatial intelligence: the next frontier beyond text and images
Fei-Fei Li frames World Labs’ core thesis: AI’s next leap is “spatial intelligence,” the ability to understand, generate, and reason about 3D spaces. This expands AI from language/vision into interacting with physical and virtual environments.
- •Defines spatial intelligence as generating, understanding, reasoning, and interacting with spaces
- •Positions large world models as the key enabling approach
- •Signals that spatial intelligence spans both virtual creation and physical-world interaction
- 1:08 – 3:42
World Labs’ mission: world models for virtual and physical “multiverse” use cases
Fei-Fei explains World Labs as a frontier model lab focused on building large world models. She emphasizes that acting in space isn’t only robotics—creative industries also need controllable, explorable 3D worlds.
- •World Labs is a two-year-old startup building large world models
- •Spatial intelligence applies to VFX, gaming, design, and more—not just robots
- •Long-term view: enable developers/builders to act within many “spaces” (a multiverse)
- •Robotics is a profoundly important application of spatial intelligence
- 3:42 – 5:16
SceniX origin story: a pragmatic real-to-sim-to-real pipeline
Yunzhu Li introduces SceniX and his research background, then lays out the company’s aim: solve robotics’ training and evaluation bottlenecks by mapping real environments into aligned digital worlds. The goal is to generate scalable data and testing inside simulation while preserving real-world fidelity.
- •Yunzhu’s North Star: robots that work in real environments
- •Robotics bottlenecks: lack of training data and slow/expensive evaluation
- •Real-to-sim-to-real: digitize real spaces so sim outcomes match real outcomes
- •SceniX team strength spans robot learning, simulation, and rendering
- 5:16 – 7:13
Marble: World Labs’ 3D generative base model meets robotics’ data bottleneck
Fei-Fei describes how SceniX first engaged World Labs as a Marble customer, which revealed deep product synergy. Marble turns prompts (text/images) into geometrically consistent 3D worlds, aligning with the need to scale data for robotics where internet-scale datasets don’t exist.
- •SceniX initially came to World Labs as a Marble customer (not via prior relationship)
- •Marble: a base model that generates geometrically consistent 3D scenes (e.g., mesh/Gaussian splats) from prompts
- •Robotics differs from LLMs: data is scarce for training and evaluation
- •Unlocking robotics scaling laws requires new sources of high-quality world data
- 7:13 – 10:56
Why the teams fit: complementary strengths across robotics, simulation, and generative 3D
They break down how SceniX’s robotics/simulation expertise complements World Labs’ generative modeling and 3D reconstruction strengths. Yunzhu explains why joining forces accelerates SceniX’s ability to move from heavy dense reconstruction toward more efficient modeling using World Labs capabilities.
- •SceniX co-founders bring deep robotics + world-class simulation (including VFX/industry experience)
- •World Labs brings strong generative models and 3D computer vision/reconstruction
- •Shared mission, minimal overlap: complementary stacks rather than duplication
- •SceniX dense reconstruction can be made more efficient by leveraging Marble-style generation/sparse modeling
- 10:56 – 12:35
What a “foundation model for robotics” could be: multimodal + action-conditioned
The discussion turns to whether World Labs will build a robotics foundation model. Fei-Fei and Yunzhu describe omni/multimodal models where “actions” are first-class: actions as inputs resemble forward simulation; actions as outputs resemble policy models.
- •Robotics foundation models likely require action outputs in addition to world-state outputs
- •Multimodal inputs: text, images, depth, and other sensors
- •Actions as input → forward model predicting world change; actions as output → policy for goal-directed behavior
- •Such base models could be fine-tuned for specific, reliability-critical deployments
- 12:35 – 18:12
Video-model approaches vs consistent 3D worlds: why consistency and interaction matter
Yunzhu contrasts popular “video model” approaches with their emphasis on building consistent worlds across space, time, viewpoints, and interactions. He argues that robots need stable causal structure—e.g., objects shouldn’t disappear mid-interaction—so simulation and world modeling must capture essential structure, not just pixels.
- •Robots require world consistency over time/viewpoint and during interaction
- •Pure video prediction can fail at interaction consistency (e.g., objects vanishing)
- •Hybrid approach: simulation-driven structure + learning-driven scaling via a data flywheel
- •Goal: capture the essential structure of tasks, then improve with more real-world data
- 18:12 – 23:01
Why simulation is essential: counterfactuals, robustness coverage, and faster-than-human training
They make the case that simulation is not optional: it enables counterfactual reasoning and scalable coverage of rare or dangerous states. Examples include self-driving (Waymo’s large-scale simulation) and benefits for robotics reliability (systematic variation) and efficiency (training at accelerated speeds).
- •Simulation enables counterfactual reasoning that real data can’t easily provide
- •Industry precedent: autonomous driving uses massive simulation hours
- •Reliability: systematically randomize lighting, friction, geometry, objects to cover state space
- •Efficiency: teleop data is slow; simulation can train beyond human-speed behaviors
- 23:01 – 29:17
Customer value today: evaluation speedups and controllable training data (without building the robot)
Yunzhu details two primary customer use cases: evaluation and training. The platform provides digital environments aligned to reality, enabling faster iteration and safer, scalable benchmarking—positioned as infrastructure that is model-agnostic and embodiment-agnostic rather than a robot hardware company.
- •Evaluation: measure performance at checkpoints (e.g., 95% vs 99.9%) quickly and safely
- •Real-world eval is slow, costly, and dangerous; simulation accelerates iteration dramatically
- •Training: controllability over parameters/distributions builds robustness confidence
- •Platform is environment infrastructure—customers bring their own robots and models
- •World Labs already saw inbound robotics demand for Marble-driven world generation
- 29:17 – 36:34
Humanoids vs near-term deployment: semi-structured environments and realistic timelines
They argue robotics adoption typically moves from structured to semi-structured to unstructured settings, with homes being the grand challenge. The conversation also addresses economic and power-efficiency realities: achieving human-like efficiency for general tasks is likely far off, requiring “measured optimism.”
- •Near-term focus: semi-structured environments (warehouses, hospitality) before homes
- •Humanoids are optimized for unstructured worlds but are hardest to solve and not always economically optimal
- •Robotics success is a system problem: hardware, software, contact dynamics, and iteration cycles
- •Human brain/body efficiency remains a high bar; predictions should be calibrated
- 36:34 – 42:20
Integration and execution plan: thoughtful merge, bi-coastal footprint, and 2-year success criteria
Fei-Fei outlines a deliberate integration strategy—leveraging Marble immediately while not rushing a full codebase/team merge. They discuss geography (SF + NY) and define success as validated lighthouse customers in key verticals, with flexible engagement for robotics companies at any stage.
- •Integration approach: collaborate and integrate gradually rather than immediately “blending everything”
- •Bi-coastal setup: SF HQ plus NY presence to attract talent and support robotics work
- •Two-year success: a small set of verticals with validated customers and clear automation value
- •Customer engagement is flexible: real-to-sim only or full real-to-sim-to-real pipeline
- •Call-to-action: it’s not too early or too late for robotics teams to engage World Labs
