By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
a16zWhat It Takes to Build an AI That Actually Knows You
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howCHAPTERS
Jump to publisher timestampsChapter summaries
Why your computer already “knows” you: tapping ambient context
Ken Liu opens by arguing that people underestimate how much actionable context is already present on a computer. He frames a future where agents can fix or act on things with minimal explanation because the system can infer what’s happening from what you see and do.
- •Computers already capture rich signals about what you’re doing, reducing the need to explain context to agents
- •Simple interactions like a screenshot plus “Fix” can be enough when the system understands what it’s looking at
- •The central question becomes whether agents can continuously analyze this background context effectively
First Person Labs and “Second”: building a model of a person
Ken Liu explains First Person Labs’ focus: creating a model of a person and putting that person “in control.” He defines the user model as more than memories—covering priorities, preferences, goals, values, commitments, and intent.
- •First Person Labs aims to create a person-model that reflects how someone thinks and what they care about
- •The model includes memories plus priorities, preferences, goals, values, commitments, and intent
- •The product concept centers on user control via this persistent personal model
Portable context for existing agents: how Second works with Grok Bot
Martin Casado probes how First Person Labs fits alongside tools like Grok Bot. Ken describes Second as “portable context” that other agents can query so tasks like booking travel can be executed with less back-and-forth.
- •Second can be queried by other agents to figure out what the user wants
- •The user can issue high-level prompts (e.g., “buy me a flight”) because preferences and constraints are already captured
- •The goal is to eliminate repetitive clarification rounds by sharing a built-up user model instantly
From baseline artifacts to “Alfred”: representation without revealing identity
The conversation explores whether Second becomes a “first person” agent or stays as infrastructure. Ken explains that a user model evolves from baseline artifacts over time, and uses an Alfred/Batman analogy to show how an intermediary can act on your behalf without exposing you.
- •User models start from baseline artifacts and build understanding over time from behavior and communication patterns
- •Second is envisioned evolving from portable context into a richer representation of the user
- •The Alfred analogy: an agent can fulfill preferences and actions without revealing sensitive identity details
The hard systems problem: capturing the raw stream of digital life
Martin highlights that the striking part is Second’s access to many channels and device context, which is a difficult systems problem. Ken describes an always-on pipeline analyzing an ‘infinite stream of events’ from OS-level signals and app usage to infer higher-level intent.
- •Second observes continuous OS and app events (switching apps, typing patterns, website pauses, connectivity)
- •Background agents build hierarchical interpretations (e.g., recognizing messaging vs. research vs. document writing)
- •Connectors plus event-stream analysis can yield strong understanding of what the user is trying to do
Proactive AI beyond notifications: reactive vs. reflective proactivity
Joel and Martin describe Second surfacing timely, useful updates without being asked. Ken breaks proactivity into levels: easy ‘reactive’ triggers versus harder ‘reflective’ behaviors that connect dots over long horizons to intervene meaningfully.
- •Users report Second proactively updating them based on evolving conversations and conditions without explicit requests
- •Reactive proactivity can become noisy (multiple agents pinging on obvious triggers)
- •Reflective proactivity connects long-term goals, prior statements, priorities, and preferences to time interventions
- •Ken argues the bottleneck is that people currently have to ask—whereas the system should anticipate needs
One trusted vault: why there should be a single Second (plus ephemeral sub-agents)
Martin asks whether users will need multiple “Seconds.” Ken argues the current proliferation of personal agents is temporary; agents will specialize, but Second should remain the consistent, trusted store—a ‘vault’—that ephemeral sub-agents can access and then disappear.
- •Specialized agents will exist for tasks, but a consistent personal context store provides the enduring value
- •Second is intended as the trusted vault holding ‘life context’ and aligning with the user’s interests
- •Users can spin up ephemeral sub-agents, but the persistent store remains singular
- •The design aim: a stable, trustworthy hub that interfaces with the software (and potentially hardware) world
Trust and security model: data, inference, and confidential computing
Joel raises the instinctive discomfort of a system that ‘sees everything.’ Ken explains the architecture in terms of data and inference, offering options like bringing your own tokens for local analysis and using confidential inference with hardware-backed guarantees rather than policy-only promises.
- •Ken frames AI security around controlling where data goes and what inference is allowed
- •Users can bring their own tokens to run analysis locally on laptop data
- •Second can offer an inference provider using trusted execution / confidential inference
- •Confidential inference is positioned as a provable hardware-backed guarantee, contrasted with policy-based promises
Storage options: local-first vs encrypted cloud with secure decryption
Martin asks how data is stored and whether it’s mixed in a shared cloud bucket. Ken outlines two approaches: keeping data local (“Second Local”) or backing it up encrypted in the cloud, with decryption occurring only under user control or inside attested secure environments.
- •Second Local keeps data on-device, only sending it to the chosen inference provider
- •A cloud-hosting option can encrypt locally before uploading, similar to password manager patterns
- •Decryption can happen ephemerally in memory or inside an attested confidential container
- •The goal is to avoid multi-tenant intermingling and maintain a secure chain end-to-end
Can an LLM keep secrets? Deterministic backstops vs reasoning-based control
Martin challenges how Second avoids divulging sensitive information under adversarial prompting. Ken distinguishes deterministic safeguards (must always be implemented) from reasoning-based guarantees that depend on model capability and contextual understanding, noting prompt injection is not definitively solvable.
- •Deterministic controls are necessary for certain categories (e.g., never retaining certain secrets)
- •Reasoning-based controls rely on context and the model’s ability to judge what to share
- •Second may model normal interaction patterns to detect anomalies (e.g., unusual message timing suggesting impersonation)
- •Prompt injection is acknowledged as a hard problem that can’t be definitively solved, improving with model capability
Cost, portability, and the target user: always-on context is expensive (for now)
Martin flags that continuously analyzing a user’s life seems expensive and asks who it’s for. Ken provides concrete token-cost examples and argues early users will be busy professionals, while falling token prices and model swapping should broaden access over time.
- •Ken estimates 24/7 multi-agent analysis can be extremely costly depending on model pricing
- •He contrasts a high-cost scenario with a much cheaper alternative model, emphasizing variability
- •Initial target users are people who need constant context and switch tools frequently
- •Token prices are dropping; portability enables swapping in new, more capable, cheaper models
Surprising use cases: coding memory, screenshot-to-fix, and cross-device context continuity
Martin asks for unexpected, useful applications. Ken highlights development workflows where Second acts as durable memory so you don’t need to re-explain context, and describes ‘context continuity’—continuing the same work seamlessly from laptop to phone.
- •In development, Second reduces the need to restate context to coding agents
- •A screenshot plus “Fix” can work because Second knows what you were doing and what the artifact is
- •Context continuity keeps work state current across devices, enabling seamless continuation
- •Ken says use cases are early and require further application building and interaction design
Mix of models and a “harness”: matching tasks to model capability and cost
Joel asks whether frontier models meaningfully outperform cheaper ones for Second’s tasks. Ken explains a hierarchy: many semantic detections don’t need expensive models, while deeper inference about goals does—leading to a mixture-of-models strategy and a system for choosing which model runs when.
- •Different tasks require different model strength: simple event interpretation vs deeper goal inference
- •Expensive models may be used less frequently for high-level reasoning, cheaper models for routine parsing
- •Second is built to be portable across models to support future local inference and rapid swapping
- •Ken describes a ‘harness design problem’—selecting models dynamically for cost efficiency
How users interact today: install app, permissions, background agents, and personal APIs
Martin asks what a new user gets and how the interface works. Ken describes installing an app, granting permissions, optionally setting tokens, and letting background agents build usable outputs—APIs representing recent actions, upcoming priorities, reminders, and personal ‘wiki’ knowledge.
- •Setup involves installing an app, granting permissions, and specifying tokens
- •Second uses both raw computer event streams and connectors to learn faster
- •Background agents run continuously, requiring little ongoing user effort
- •Outputs are exposed as APIs (recent activity, next-week priorities, reminders, people/processes) that other agents and apps can query
Toward Jarvis and ubiquitous access: cloud MCP now, apps everywhere later
The conversation turns to embodiment and ubiquitous computing—why AI with context isn’t ‘everywhere’ yet. Ken says OS constraints limit full deployment today, but users can access Second via internet-connected agents (e.g., through Grok Bot calling Second’s API), with broader multi-device apps and possibly devices in the future.
- •Martin wants a more embodied, always-available interface; Ken references the ‘Jarvis’ idea as ubiquity beyond intelligence
- •Ken says current OS lockdown limits what can be done on some devices today
- •As of now, Second can be used via cloud MCP: any agent with internet that can call the API can access it
- •They discuss practical workflows like using Grok Bot audio to converse with Second while walking or driving
Fort Knox meets delightful UX: focusing on infrastructure, minimal surfaces, and rollout
Martin asks how the company balances hardcore security infrastructure with a smooth user experience. Ken says the work splits into infrastructure and interaction problems; after struggling with UI complexity, they narrowed surfaces to texting and APIs to keep interactions smooth while prioritizing secure, responsible scaling.
- •Ken separates the challenge into infrastructure vs interaction design, noting they’re very different
- •They made missteps trying to go all-in on interaction/UI and found it hard to get ‘delightful’ details right
- •They chose to minimize product surfaces: primarily texting and APIs, with optional laptop co-work capabilities
- •Ken closes by inviting listeners to try it and notes the rollout is paced to scale inference and keep data secure
Publisher timestamps
- 0:00Intro
- 0:50What Is First Person Labs?
- 2:43The Alfred Model: Your Agent's Butler
- 3:56What Your Computer Already Knows About You
- 5:59Proactive AI: Reactive vs. Reflective
- 8:16Why You Only Need One Second
- 10:27Can You Trust Something That Sees Everything?
- 15:41Can an LLM Keep Your Secrets?
- 17:40Cost, Models, and the Target User
- 20:20Use Cases and the Model Mix
- 23:34How You Use Second (APIs and Apps)
- 26:20Toward Jarvis: AI Everywhere
- 30:43Fort Knox Meets Delightful UX