CHAPTERS
- 0:00 – 2:02
Devi Parikh’s path into computer vision: from “pattern recognition” to research joy
Sarah opens by introducing Devi Parikh and her work at Meta and Georgia Tech, then asks about Devi’s background and how she found her way into AI. Devi recounts moving from India to the U.S., discovering research as an undergrad, and how that early enjoyment shaped her trajectory.
- •Moved from India to the U.S. after high school; undergrad at Rowan University
- •Early exposure to research via “pattern recognition” projects and mentorship
- •Initial uncertainty about pursuing a PhD vs. a master’s thesis track
- •Interest in work that feels joyful and intrinsically motivating
- 2:02 – 2:39
Choosing a thesis direction: why vision ‘shows you’ what the model is doing
Devi explains how she gravitated toward computer vision during her time at CMU. She highlights the appeal of visual outputs compared to purely metric-driven feedback, which helped define her thesis direction over time.
- •Transition from non-visual projects to image processing / computer vision
- •Motivation: qualitative inspection of outputs vs. abstract metrics
- •Computer vision’s immediacy helps diagnose what works and what doesn’t
- •Thesis topic evolved gradually rather than being fixed upfront
- 2:39 – 5:12
A unifying theme: richer human–machine interaction across modalities
Looking back, Devi identifies a consistent thread in her work: enabling more meaningful interaction between humans and AI systems. That led from visual attributes and interpretability to full natural-language interfaces like VQA and captioning, and eventually to AI as a creativity tool.
- •Core theme: human–AI interaction and communication
- •Attributes/adjectives as a bridge for teaching concepts and interpretability
- •Shift into vision-and-language tasks (VQA, captioning) as interaction deepens
- •Expansion into ‘AI for creativity’ as a next step in expressive interfaces
- 5:12 – 6:29
Why Meta/FAIR: an academic–industry split that turned into a long-term home
Devi describes joining FAIR for what was intended to be a one-year stint during a faculty transition. The collaboration worked well enough that she continued for years, splitting time between teaching at Georgia Tech and research at Meta.
- •Joined FAIR during Virginia Tech → Georgia Tech transition
- •Existing network from Microsoft Research connections and internships
- •Planned one-year visit extended into multi-year arrangement
- •Split schedule: teaching in Atlanta, research in Menlo Park
- 6:29 – 7:53
Meta’s generative AI push: building across text, images, video, 3D, audio, music
Devi outlines the newly formed generative AI organization at Meta and the modalities it spans. She frames the opportunity as shifting from pure content consumption toward enabling many more people to create what they imagine across products and surfaces.
- •New org spanning LLMs, image/video generation, 3D, audio, music
- •Generative tools as complements to (not replacements for) consumption
- •‘Create what’s in your head’ as a broad product and UX motivation
- •Applies across many surfaces: search, expression, communication
- 7:53 – 10:29
Make-A-Video origin and core idea: leverage image priors, learn motion separately
Elad prompts a deep dive into Make-A-Video. Devi explains the project’s timing (post DALL·E 1 era) and the central strategy: reuse advances in text-to-image diffusion to learn appearance and language alignment, while using video to learn motion and temporal coherence.
- •Video generation emerged as the next open frontier after image breakthroughs
- •Key decomposition: appearance/text alignment vs. motion dynamics
- •Benefits: less to learn from scratch; inherit concept diversity from images
- •Avoids needing paired video+text; use image-text pairs plus unlabeled video for motion
- 10:29 – 12:43
How Make-A-Video trains: from independent frames to temporally coherent clips
Devi explains the training behavior concretely: initialize from an image generator that can produce multiple frames that don’t match over time, then train using video data to enforce temporal consistency. She also discusses generalization via learned notions of “flying,” combining visual concepts with motion patterns.
- •Initialization produces plausible but temporally incoherent frames
- •Training objective pushes frames to become consistent—turning images into video
- •Generalization intuition: combine ‘corgi appearance’ with ‘flying motion’ learned elsewhere
- •Acknowledges interpretability limits in large deep architectures
- 12:43 – 15:02
What’s missing for real video: longer narratives, memory, transitions, object persistence
Devi sets expectations: current systems resemble short animated images rather than true storytelling video. The frontier includes longer duration, richer events (appear/disappear), scene transitions, and memory so reappearing objects remain consistent.
- •Current outputs: ~4 seconds, mostly one scene with small motion
- •Major gaps: scene transitions, objects entering/exiting/reappearing
- •Need persistent identity and consistency across time (memory)
- •Goal: more complex stories rather than single-shot animations
- 15:02 – 19:09
Why progress is slower than images: infra cost, video representations, hierarchy, and data recipes
Devi explains why step-change improvements haven’t appeared as quickly in video as in images. She cites slower iteration due to scale, the lack of efficient representations for redundancy, architectural challenges for hierarchical long-range structure, and the immaturity of video-focused data curricula.
- •Infrastructure: storage/compute makes experimentation and generation slower
- •Representation: redundancy across frames not exploited well; frames treated too independently
- •Architecture: long videos require hierarchical structure, memory, and transitions
- •Data: need better ‘recipes’/curricula (from simple to complex), not just more sources
- 19:09 – 20:43
Video and robotics: embodied feedback loops change how visual streams should be modeled
Asked about robotics, Devi emphasizes video understanding over video generation. She highlights embodiment: an agent’s actions determine the next observations, creating a feedback loop that differs from passive video viewing and can be exploited for better learning and prediction.
- •Robotics relevance primarily via video understanding, not generation
- •Embodied agents perceive inherently temporal visual input
- •Actions causally influence future frames—useful learning signal
- •Contrast with disembodied vision where the stream is uncontrolled
- 20:43 – 23:32
Control in video generation: multimodal prompting and iterative editing over ‘prompt-and-pray’
Devi argues that creative tools require controllability—text prompting is only a first step. She outlines two directions: richer, multimodal inputs (sketches, reference images, audio clips, seed videos) and iterative editing interfaces that let users refine outputs predictably over multiple turns.
- •Creative expression demands reliable control aligned with user intent
- •Text prompts beat random sampling but remain indirect and brittle
- •Multimodal prompts: sketches/images/audio/video as constraints and guidance
- •Iterative editing: communicate what to change rather than rerolling generations
- 23:32 – 24:53
Control will lag capability: first make ‘random good,’ then scale prompting and editing (plus video-to-video products)
Devi predicts that better control typically follows once base generation quality is strong. She distinguishes generating from scratch (needs strong iterative control) from editing existing videos, where products are already emerging (e.g., stylization workflows).
- •Historical pattern: capability improves before fine-grained control catches up
- •Near term: progress in core text-to-video quality and reliability
- •Separate track: video-to-video editing/stylization may mature faster in products
- •Interfaces matter: intuitive editing beats hours of tool learning
- 24:53 – 27:19
Audio’s role: expressive soundtracks, underinvestment, and hard compositionality problems
The discussion shifts to audio generation and how it enriches visual media. Devi notes current text-to-audio quality is uneven and limited in duration, and highlights open challenges like sequencing and simultaneous superimposed sounds, plus the broader underinvestment in audio and music compared to visuals.
- •Text-to-audio: typically seconds to tens of seconds; success is inconsistent
- •Audio makes visual content more delightful and expressive but is underinvested
- •Compositionality remains hard: sequences of events and overlapping sounds
- •Parallel to video: longer duration and structured control are still unsolved
- 27:19 – 34:34
Use cases, creators, and the future of social media: from democratized art to AI agents in the social graph
Devi and the hosts explore where generative media may show up first and how it changes expression online. They discuss skepticism about ‘who creates,’ countered by social platforms that lowered creation friction, and Devi predicts broad deployment—plus new social dynamics as AI agents become participants.
- •Near-term: generation as a better way to find/express what you mean (vs. searching)
- •Democratization parallels: Instagram/TikTok reduced friction; Midjourney shows demand
- •Artists span a spectrum: AI as tool vs. collaborator in the creative process
- •Social shift: generative tools at scale; AI agents may become entities in social networks
- 34:34 – 39:50
Rapid-fire advice: underexplored research areas, academia in the era of big models, and ‘don’t self-select’
In closing, Devi highlights research gaps—especially control and deeper multimodality across generation and understanding. She shares what she enjoyed at CVPR (meta discussions about big models and academia), offers time-management tactics, and ends with a career maxim: apply and let the world decide.
- •Underexplored: control mechanisms and true cross-modality systems that ingest/generate many modalities
- •CVPR focus: vision-language-creativity; inspired by ‘Scholars and Big Models’ workshop
- •Productivity tip: schedule tasks on a calendar (not just a to-do list) to force time realism
- •Career advice: ‘Don’t self-select’—apply for roles/fellowships/internships and let others say no
