Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

a16za16z

What It Takes to Build an AI That Actually Knows You

AI agents are getting smarter, but every new agent still has to learn who you are. What if your preferences, priorities, relationships, and digital context could move with you instead?

Ken LiuguestMartin CasadohostJoel de la Garzahost
Nov 18, 202633mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

At a glance

WHAT IT’S REALLY ABOUT

Second: a secure, always-on personal context layer for AI agents

  1. Ken Liu introduces First Person Labs’ “Second” as a persistent model of a person—capturing memories, preferences, priorities, and intent—that other agents can consult as portable context.
  2. The system draws on an always-on stream of computer activity plus connectors (e.g., messaging apps) to infer what a person is doing and what might matter to them, enabling proactive updates without being explicitly asked.
  3. The conversation focuses heavily on security and trust, including local-only operation with user-supplied tokens, cloud options with encryption, and confidential inference using hardware-based guarantees rather than policy promises like “zero data retention.”
  4. Cost is presented as a key constraint for 24/7 analysis, pushing the product toward a mixture-of-models approach and a tunable tradeoff between continuous background processing and slower, query-time reasoning over raw logs.
  5. Long-term, the speakers imagine Second as a ubiquitous layer across devices—closer to “Jarvis” continuity—while today it is delivered as an installed app that generates APIs and supports interaction via text and agent integrations.

IDEAS WORTH REMEMBERING

5 ideas

Second is positioned as a personal “user model” that other agents can query.

Ken Liu describes Second as a system that accumulates a person’s memories, priorities, preferences, goals, values, commitments, and intent, then makes that representation available to other agents on demand.

A major premise is that your computer already contains rich, underused behavioral context.

Ken Liu says modern operating systems already expose a continuous stream of events (app switching, pauses on websites, typing patterns, Wi‑Fi connections), and Second applies layers of agents to synthesize higher-level meaning from that raw stream.

The product aim is reflective proactivity, not just more notifications.

Ken Liu distinguishes “reactive” proactivity (triggered by obvious new events like incoming emails) from “reflective” proactivity that connects dots across many events and long time horizons (e.g., months-old statements and current priorities).

Trust is treated as an architecture problem: control over inference plus provable isolation.

Ken Liu emphasizes that because Second is ambient and always-on, users should be able to choose where inference happens, including using their own tokens for local analysis; he also discusses confidential inference where hardware-based guarantees limit who can see data during inference.

Continuous personalization is capability- and cost-constrained, requiring tradeoffs.

Ken Liu says the system is expensive if you continuously analyze traces with multiple agents, so early target users are likely busy professionals; he also describes a cost–fidelity tradeoff between deeper continuous “cache time” analysis versus using raw logs and paying more at query time.

WORDS WORTH SAVING

5 quotes

So we focus on essentially creating a model of a person and then put that person in control.

— Ken Liu

I think people are very underestimating, like, how much the computer already knows.

— Ken Liu

Reactive proactivity is, like, straightforward, easy to do, but what's harder is what I call reflective ones, which are things that connect the dots over, like, uh, many events and long time horizon, right?

— Ken Liu

It's like a vault.

— Ken Liu

There's only two things when it comes to AI. There's data, there's inference, right?

— Ken Liu

Read the available transcript
First Person Labs and “Second” conceptUser model: memories, preferences, goals, valuesAmbient computer event stream analysisConnectors to channels (messages, chat, email-like data)Proactive AI: reactive vs. reflectiveTrust architecture: local tokens, confidential inference, hardware attestationData storage modes: local-only vs encrypted cloud with secure decryption environments, model portability and mixture-of-models cost strategy

Ask ChatGPT about this episode

Get its key ideas, plus a summary of each chapter linked to that point in the video, right in your chat.

  1. Add uListen AI to ChatGPT.

  2. Paste this prompt into a new chat.

    Use uListen AI to give me this episode's key ideas and a chapter-by-chapter summary, with timestamps: https://www.youtube.com/watch?v=kBJe4H_WobA

SPEAKERS

  • Ken Liu

    guest

    Founder at First Person Labs, building the “Second” personalized AI assistant and its privacy/security architecture.

  • Martin Casado

    host

    a16z general partner focused on infrastructure/enterprise tech, interviewing Ken about First Person Labs and its product design.

  • Joel de la Garza

    host

    a16z host/co-host with a security/systems focus, asking detailed questions about confidential inference and data handling.

High quality AI-generated summary created from speaker-labeled transcript.

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.