Stanford Online

Stanford CS153 Frontier Systems | Amit Jain from Luma AI on Unified Intelligence Systems

For more information about Stanford's online Artificial Intelligence programs, visit: https://stanford.io/ai Follow along with the course schedule and syllabus, visit: https://cs153.stanford.edu/ In week three of CS153, the instructor hosts Amit Jain from Luma to discuss “Unified Intelligence Systems” as a follow-up to a prior lecture on visual intelligence. Jain recounts his Apple work on LiDAR for projects including Titan and Vision Pro, and how early exploration of generative models and differentiable 3D led to founding Luma with an initial focus on large-scale 3D capture. Luma then shifted to generative video in 2023 to leverage the scale of internet video data, releasing the Dream Machine model in March 2024 and rapidly reaching millions of users, while building preference-based feedback loops and human annotation pipelines. Jain explains Luma’s multimodal AI factory—pretraining, post-training, deployment, and reinforcement learning—its security constraints for studio clients, and a move toward unified transformer architectures that jointly reason across text, images, video, and audio to enable end-to-end creative and professional workflows. Guest speaker: Amit Jain is the CEO and co-founder of Luma AI, a research lab developing multimodal foundation models aimed at "unified intelligence." Under his leadership, Luma has scaled from a 3D-capture pioneer into a leader in generative video, raising a $900M Series C following the success of its Dream Machine and Ray video-reasoning models. By 2026, he has steered the company into large-scale infrastructure projects including Project Halo — a 2-gigawatt AI supercluster — to build the next generation of "world models" capable of simulating physical reality. He founded Luma in 2022 from Apple, where he was a Systems and Machine Learning Engineer. At Apple, he led development of the Passthrough feature for Apple Vision Pro and was instrumental in integrating the first LiDAR sensors into the iPhone — foundational work for modern spatial computing. His background also includes physics and mathematical simulation. Follow the playlist: https://youtube.com/playlist?list=PLoROMvodv4rN447WKQ5oz_YdYbS74M5IA&si=DOJ5amlyRdyMJBhG

CICS153 InstructorhostAJAmit Jainguest

May 6, 2026·57m·Watch on YouTube ↗

📖 CHAPTERS

1. 1

   Why Luma exists: from Apple LiDAR to a “world simulator” thesis

   The instructor introduces Amit Jain and traces Luma’s origin to Amit’s Apple work on LiDAR and next-generation interfaces. Amit explains how early generative-model exploration (post-NeRF, pre-DALL·E) led to the idea that differentiable learning across modalities could enable world understanding and generation.

2. 2

   What “differentiable world learning” means in practice

   Amit defines differentiability as the prerequisite for training via gradient descent and explains why it is the core tool of the current deep learning era. The discussion connects differentiability to the ability to optimize models at scale using compute and data.

3. 3

   First product flywheel: 3D capture app and the scale problem

   Luma’s early approach was to build a massive 3D dataset via a consumer capture app that productionized NeRFs and Gaussian splats. The team quickly learned that even a popular app couldn’t reach the data scale needed to learn “enough about the universe.”

4. 4

   Pivot to generative video: why Hopper changed the roadmap

   After confronting data scale, Luma redirected efforts toward video as a learnable proxy for 3D understanding over time. The NVIDIA Hopper era made the compute economics feel feasible, leading to infrastructure investments and the release of Dream Machine in 2024.

5. 5

   Bootstrapping Dream Machine’s feedback loop (and what went wrong initially)

   Amit explains how Luma extracted preference signals from user behavior (likes/downloads) to find the “narrow band” of outputs people value. Early signals were noisy—some users downloaded bad videos to mock AI—forcing Luma to build human filtering and labeling operations.

6. 6

   From model to product: capturing interaction data and continual improvement

   The conversation highlights how product design becomes part of the training pipeline. Luma increasingly learns from granular interaction traces in its agent interface to improve models via post-training and continual reinforcement loops.

7. 7

   Why “unified intelligence” is needed beyond video alone

   Amit argues that video by itself lacks logic, causality, and “why” something matters—limitations that appear in creative and robotics workflows. Luma reframes the target as unified multimodal intelligence that combines language-level reasoning with physical/world understanding.

8. 8

   Inside the Luma “factory”: pre-training, scaling, and post-training at production scale

   Amit walks through Luma’s version of the frontier AI pipeline and the design constraints behind it. He describes multimodal pretraining at large data scale, training infrastructure on H100s (and future GB300s), and heavy post-training using customer and preference data.

9. 9

   Enterprise deployment constraints: studios, confidentiality, and learning from traces

   The instructor probes “mission-critical context” where customers require strict isolation. Amit outlines internal controls and policies to prevent sensitive project data from entering training while still learning from non-content interaction traces.

10. 10

    Unified models in action: generating these slides end-to-end

    Amit demonstrates unified intelligence through a real workflow: providing a reference slide style, a mind-map scaffold, and instructions to generate polished slides in one shot. He uses this to argue that pixels and words are both carriers of “intelligence” when structured meaningfully.

11. 11

    Why fused systems weren’t enough: bridging the understanding–generation chasm

    The class compares unified models to earlier fused VLM approaches (e.g., separate language and diffusion components connected by a thin bridge). Amit argues that those systems can’t reliably produce structured visuals (like schematics) because reasoning and generation aren’t truly integrated.

12. 12

    Luma’s unified architecture: one backbone to reason across modalities

    Amit outlines Luma’s bet on a shared transformer backbone that encodes different modalities into a common space for joint reasoning, analogous to encoders feeding a shared “neocortex.” He notes it took many failed attempts but claims confidence scaling to very large models.

13. 13

    Agents as the computer of the future: REPL loops, skills, and tool harnesses

    The discussion shifts from model architecture to deployment architecture—how agents perform iterative work using a REPL-like loop. Luma bets on “mega models” coupled with external tools and a thick layer of reusable skills (playbooks) to execute tasks end-to-end.

14. 14

    Business, capital intensity, and adoption: why creatives are leaning in

    Amit explains why multimodal systems can become more compute/data intensive than language over time, while still being a strong near-term business in creative markets. He describes rapid creative-industry adoption as quality improves, shifting sentiment from fear to productivity and exploration.

15. 15

    Q&A lightning round: Sora shutdown, copyright, GANs vs diffusion, and what’s next

    Amit hypothesizes that OpenAI’s Sora pause reflects organizational focus constraints rather than market size, and he argues copyright law remains orthogonal to generation capability. He also discusses why GANs persist in niches, why diffusion may be “on the way out,” and what’s needed for video models to become as useful as LLMs.