No Priors Ep.61 | OpenAI's Sora Leaders Aditya Ramesh, Tim Brooks and Bill Peebles

By learning to predict future frames from raw video, Sora implicitly acquires knowledge of 3D structure, physics, and social interactions—similar to how humans mentally simulate scenarios—making it a plausible building block for AGI.

Instead of extending image models, the team designed Sora from scratch around diffusion transformers and spacetime patches, optimized from day one for minute‑long HD video, diverse aspect ratios, and broad data coverage.

Using GPT‑style transformers on video tokens, they observe that more compute and data systematically improve quality, suggesting a clear path to stronger physical reasoning, longer sequences, and richer interactions as resources increase.

Representing videos as 3D cubes in latent space lets a single model handle different resolutions, durations, orientations, and even still images, analogous to how text tokens let LLMs train on diverse textual formats.

Early artist tests show Sora lowering barriers to high‑quality visual storytelling and hint at entirely new interaction paradigms (e.g., personalized stories, dynamic educational content) beyond traditional film or clips.

OpenAI is limiting access to artists and red teamers while it addresses misuse risks (deepfakes, misinformation, sensitive content) and works to make generation cheaper and faster enough for mainstream use.

While Sora can maintain some state (e.g., a bite mark in a burger, paint trails), it can still lose track of objects or break physical consistency over time, signaling a key research frontier for future versions.