Lex Fridman PodcastMark Zuckerberg: First Interview in the Metaverse | Lex Fridman Podcast #398
CHAPTERS
- 0:00 – 2:20
Photorealistic avatars and the feeling of presence (Lex meets Mark in VR)
Lex opens by explaining that the interview happens inside the metaverse with photorealistic Codec (Kodiak) Avatars and spatial audio. They react in real time to how intimate and “in-the-same-room” it feels, including the subtle social awkwardness of personal space in VR.
- •Lex frames this as the future of meaningful online connection
- •Immediate reactions to realism, lighting control, and spatial closeness
- •Presence and emotional impact compared to normal remote calls
- •Humor about jujitsu and how “real” it feels
- 2:20 – 5:17
How Kodiak/Codec Avatars work: scanning, expressions, bandwidth efficiency
Mark explains the research pipeline behind Kodiak Avatars: detailed scans of face/body across many expressions, then real-time driving of the avatar via headset sensing. The approach aims to deliver realism with better bandwidth efficiency than transmitting full video or volumetric capture.
- •High-detail scanning of many facial expressions builds a generative model
- •Headset sensors capture expression and transmit an encoded representation
- •Photorealism relies on details like wrinkles, asymmetry, and eyes
- •Presence is driven by nonverbal cues more than words
- •Goal: realism without massive video bandwidth
- 5:17 – 15:28
Making it scalable: fast phone scans, rollout strategy, and full-body tradeoffs
Lex asks how this becomes accessible beyond hours-long lab scans. Mark outlines a progression: quick smartphone scans in minutes, gradual integration into Meta apps, and engineering tradeoffs around compute cost—especially for full-body fidelity versus face fidelity.
- •Vision: 2–3 minute phone-based scan achieving near-current quality
- •Progressive rollout rather than “everyone gets it overnight”
- •Strong early fit for meetings/work presence; mixed fit for games with stylized aesthetics
- •Compute/rendering constraints shape what fidelity is practical
- •Face realism prioritized because micro-expressions carry emotional meaning
- 15:28 – 19:38
Quest 3 and mainstream mixed reality: price, sensors, and comfort leap
The conversation shifts to Quest 3 as the first mass-market mixed reality headset: improved passthrough, better chip, thinner form factor, and much lower price than Quest Pro. Mark emphasizes that each headset generation adds genuinely new capability, with MR being the big step change.
- •Quest 3 positioned as mainstream MR at ~$500 vs Quest Pro at $1,500
- •Major improvements: higher-resolution sensors, stronger chipset, better displays
- •40% thinner for comfort and longer sessions
- •Mixed reality enables overlaying digital objects/people onto the physical world
- •Mass adoption unlocks the developer flywheel and experimentation
- 19:38 – 22:28
Mixed reality interaction: passthrough alignment, AI mapping, and hand tracking demos
They dig into the technical challenge of MR passthrough—mapping camera views to each user’s eye position in real time—and why it’s an AI/compute problem, not just cameras. Hand tracking becomes a key input method, highlighted by a low-latency piano-learning demo on a tabletop.
- •Real-time perspective correction because cameras aren’t located at users’ eyes
- •Multiple cameras + 3D mapping + low latency are required for natural MR
- •MR feels safer for fitness/shooters because you can see real furniture/space
- •Hand tracking reduces dependence on controllers
- •Demonstrations like the piano trainer show precision and responsiveness
- 22:28 – 30:29
Holograms everywhere: redefining the ‘real world’ as physical + digital (and AI embodiment)
Mark lays out a near-future vision: glasses that blend the physical world with ubiquitous holograms—screens, media, games, and interactive objects that appear and disappear on demand. He also introduces a key leap: embodied AI agents joining meetings and social spaces as “participants.”
- •Future glasses as the bridge to persistent holographic content
- •‘Real world’ becomes a coherent blend of physical and digital
- •Many physical objects (screens, board games, even ping pong) may be better as holograms
- •Hybrid meetings: physical people + remote photorealistic avatars in the same room
- •AIs can be embodied as realistic avatars and join collaboration as agents
- 30:29 – 32:51
Identity and avatar authenticity: emotional tuning and ‘temporally inaccurate’ realism
They explore the subtle identity issues introduced by photorealistic avatars: do you want your avatar to be more expressive than you are, and should people control that? Mark notes a future social tension where avatars can be realistic but not up-to-date—haircuts, shaving, or “best self” capture moments.
- •Per-person tuning: how wide a smile should appear, how expressive to be
- •Avatars can represent an ‘ideal’ or more communicative version of the self
- •Photorealistic capture may not match the current moment (hair, grooming changes)
- •Raises questions about authenticity and self-perception
- •Psychological acceptance: mixing photorealistic and stylized worlds may feel normal
- 32:51 – 37:14
Grief, memory, and digital afterlife: talking to loved ones who are gone
Lex asks about the emotionally charged possibility of interacting with deceased loved ones via realistic avatars. Mark responds cautiously: it could help with grief and memory, but could also become unhealthy—requiring careful norms, research, and policy grounded in existing experiences with memorialized social media accounts.
- •Metaverse could enable powerful ‘revisiting’ of memories and interactions
- •Risk of unhealthy attachment or delayed grieving
- •Meta already navigates death-related account policies (memorialization, access limits)
- •Privacy boundaries remain critical (e.g., not exposing private messages)
- •Norms and best practices will need to evolve with realism
- 37:14 – 40:47
AI personas across Meta apps: assistants, businesses, and social/group-chat fun
Mark describes Meta’s AI strategy: many specialized AIs rather than one superintelligence. Use cases span a neutral assistant (Meta AI), customer service for businesses, and playful social AIs that enhance group chats with humor, trivia, interests, and game-like interactions.
- •Preference for many AIs with different roles vs one central superintelligence
- •Meta AI as a ‘vanilla’ factual assistant integrated into chat threads
- •Business messaging AIs: reservations, support, shopping via natural conversation
- •Social company angle: AIs that facilitate connection in group chats
- •Gaming angles: better NPCs and shared experiences like dungeon masters
- 40:47 – 51:52
From characters to creator replicas: safety, predictability, and AI Studio timelines
Lex presses on what it takes to create an AI based on a real person (e.g., Snoop Dogg or a creator). Mark distinguishes fictional characters (easier) from creator-representative AIs (harder), because the model must reliably stay within a creator’s boundaries and not say things they wouldn’t endorse; AI Studio aims to let anyone create AIs over time.
- •Fictional/role-play personas are easier because constraints are looser
- •Creator-based AIs require strong guardrails: alignment with the creator’s intent
- •Core challenge: predictability and reliability in model behavior
- •Staged rollout: experiments first, broader creator tooling later (next year for creator replicas)
- •AI Studio vision: user-generated AI creation analogous to other UGC
- 51:52 – 57:49
Open-source Llama 2: risk vs benefit, adoption surge, and Llama 3 considerations
Lex asks about Llama 2’s release and whether Llama 3 is coming. Mark argues the benefits of open-sourcing a foundation model outweigh the risks after rigorous red-teaming, and notes adoption exceeded expectations; Llama 3 (or the next foundation model) is always in progress but requires similar safety processes and productization priorities.
- •Open-sourcing foundation models can create outsized ecosystem value
- •Safety approach: red-teaming and risk assessment before release
- •Llama 2 reception: downloads and usage exceeded expectations
- •Priority: integrate Llama 2 into consumer products (personas, assistants)
- •Next model is being trained, but release depends on readiness and safety review
- 57:49 – 1:03:21
Reality, ethics, and remote work: what changes when presence breaks physics
They return to the philosophical core: if the experience feels real, in what sense is it real? Mark discusses how presence boosts intimacy, trust, and nonverbal communication, and how metaverse tech may reshape remote work by restoring the ‘togetherness’ that video calls lack, while also raising questions about harm, rules, and anonymity in digital worlds.
- •‘Real’ may be defined by subjective conscious experience, not shared location
- •Presence restores nonverbal bandwidth and relationship-building dynamics
- •Ethical boundary-setting: harm differs in digital spaces; anonymity can increase toxicity
- •Metaverse as a physics shortcut: instant ‘teleportation’ into shared spaces
- •Long-term thesis: the most human/social computing platform blends physical + digital
- 1:03:21 – 1:04:38
Wrap-up: awe, gratitude, and the first metaverse podcast experiment
Lex closes by emphasizing how repeatedly he forgets Mark isn’t physically present, calling it one of the most profound tech experiences of his life. They end with humor and appreciation for the teams building toward this blended reality future.
- •Lex reiterates the emotional intensity and intimacy of the experience
- •Mark reflects on the novelty of conveying physical presence through tech
- •Acknowledgment of engineering teams and the roadmap ahead
- •Lighthearted closing jokes about where they ‘really’ are
- •Farewell and final reaction to how ‘amazing’ it feels
