By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
a16zWhat It Takes to Build an AI That Actually Knows You
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howTranscript
40 min read · 7,657 words- 0:00 – 0:50
Intro
- KLKen Liu
People are very underestimating how much the computer already knows. You no longer have to explain anything to your coding agents. You can just screenshot anything and just say, "Fix," and then you would know what you're fixing. The question then is like, well, can you just like analyze the heck out of it?
- MCMartin Casado
What context does Grok Bot not have that First Person Labs would allow?
- KLKen Liu
One of the biggest problems right now, like people have to ask. You're not supposed to ask these things. They're super intelligence. But if you have a user model that just builds up by itself, and then these things just keep analyzing the background, then you can just share that over instantly. We focus on essentially creating a model of a person and then put that person in control.
- JGJoel de la Garza
When people think about this thing sees everything you do, there's an immediate sort of like, whoa.
- KLKen Liu
If you don't trust, let's say, us, in principle, you should be able to bring your own tokens to then analyze data that's on your laptop locally.
- MCMartin Casado
Can you talk through what you think are like the more surprising use cases that you found?
- KLKen Liu
For development, it turns out that-
- 0:50 – 2:43
What Is First Person Labs?
- JGJoel de la Garza
Thank you, Ken, for joining us. Really happy to have you on to talk about, uh, what you've been working on at First Person Labs.
- MCMartin Casado
Is this the first time you're talking about it?
- KLKen Liu
Yes.
- MCMartin Casado
Yes. So what do you do?
- KLKen Liu
So we focus on essentially creating a model of a person and then put that person in control. And the model of a person basically means that understanding their memories, understanding what they do, but more of that, uh, their priorities, preferences, goals, values, and commitments, and intent.
- MCMartin Casado
So as-- Sorry, just let's just make this very simple. I'm a big Grok Bot user, so I use Grok Bot a lot. Where would First Person Labs fit into my usage of Grok Bot?
- KLKen Liu
Yeah. The way, the way it works is that you would just bring a context over to Grok Bot. Um, it's like portable context. You go to Grok Bot and you say, "Hey, can you talk to my thing to figure out what I want?" And then the two of them will start talking, and then you can just send prompts like, oh, buy me a flight, and you would know what flight you're talking about.
- MCMartin Casado
What, what context does Grok Bot not have that First Person Labs would allow?
- KLKen Liu
I would ask-- I would probably ask the opposite question, like what, what, what does First Person Labs not have if you build a user model for yourself, right? Let's say your preferences on the seat on your airplane, the preferences of your, uh, uh, how early you would like to arrive at the airport, and, um, like where are you flying? Uh, are you flying for a class or, or, or normal? And all of that, right, as of today, you have to explain that to Grok Bot. You have like, okay, there's multiple rounds of like questioning. But if you have a user model or like a model view that just builds up by itself, and then these things just keep analyzing the background, then you can just sh-share that over, uh, instantly.
- JGJoel de la Garza
So for example, the OpenAI released two new cyber-capable models yesterday.
- KLKen Liu
Mm.
- JGJoel de la Garza
And I haven't been using my Codex in a while, but I have been using Second, so I can just go start using Codex and it has all the other work that I've already done and all the additional context already saved?
- KLKen Liu
Yeah. Yeah. Um, you just go to Codex and like, "How's my Second? What's going on?"
- JGJoel de la Garza
That's awesome.
- 2:43 – 3:56
The Alfred Model: Your Agent's Butler
- MCMartin Casado
Does Second ever be a first person? [laughs] Will it ever be a first or will it always go through agents?
- KLKen Liu
Uh, I, that's a good question. I think user model is a, it's a, it's a, it's a process. You have to start somewhere. You start with a baseline foundational list of artifacts, what you did, uh, who are the people in your life, maybe your preferences. And over time, it starts analyzing like the way you talk-
- MCMartin Casado
Yeah
- KLKen Liu
... the way you, uh, present yourself, the way you share information.
- MCMartin Casado
Yeah.
- KLKen Liu
And over time, you, you would like build up even more artifacts on top and more understanding on top, and maybe some of that you train into the model. And so, um, over time, you will go from a portable context into something more of a representation of yourself. Um, one example I like to think of is this like, um, Alfred and Batman example, where if, let's say, Batman wants to buy, buy, buy a cheeseburger and then get Alfred to do it, um, Alfred would not reveal, for example, that it's Batman doing it, but he would just buy it. He would state the preference.
- MCMartin Casado
Alfred being Batman's butler.
- KLKen Liu
Oh, yes, yes, yes.
- MCMartin Casado
Okay.
- KLKen Liu
Being the butler, yeah.
- JGJoel de la Garza
Or his agent.
- KLKen Liu
Yeah, yeah. Or his agent.
- JGJoel de la Garza
Or his agent. [laughs]
- KLKen Liu
Yeah, yeah.
- JGJoel de la Garza
I should call my agent Alfred.
- MCMartin Casado
That's a great idea.
- KLKen Liu
Yeah, yeah. He would just do it. Um, he would just do it, and then, uh, you, you would trust him because you know that they have a good model of you and, uh, what that means in, when it comes to getting things done.
- 3:56 – 5:59
What Your Computer Already Knows About You
- MCMartin Casado
It's, it's inter- it's interesting, which is like, so you de- you describe First Person, Person Lab as, as having this thing called a Second. The Second understands who you are and has context, and then other agents talk to that Second, right?
- KLKen Liu
Yeah.
- MCMartin Casado
And so, like, it kind of understands your preferences, it understands what you're doing, what you wanna do. Um, but I, I think that that actually doesn't highlight another thing that you've been working very hard on, and I think it's worth talking about, is like you're actually able to get access to context, um, digital context that a person has, like messages, chat, kind of email in a way, and make that also available. So can you talk a little bit about that? Like, so here, being a, one of the earliest First Person Lab users-
- KLKen Liu
Yeah
- MCMartin Casado
... I would say the thing that struck me was that, not that it knew who I was, but more it had access to all of my channels very well. So maybe... And then by the way, which it turns out to be a very hard systems problem, so maybe talk a little bit about that aspect of what you're doing.
- KLKen Liu
Yeah. Um, it is indeed a very hard problem, especially to get it secured private. Um, so the way it roughly works is that right now on your computer, um, there's like an infinite, an infinite stream of events. As you use your computer, you know, the apps that you switch to, the way you pause on certain websites, the way you type, and the way, like, your Wi-Fi is connected, and that the OS already gives you. Um, this is across Apple, Microsoft, and even Linux. Um, and the question then is like, well, can you just like analyze the heck out, the heck out of it, right? Like this raw stream twenty-four/seven, apply agents on top, twenty-four/seven agents. So I can first synthesize, for example, well, this piece of typing at this particular app might mean that you're sending a message. And then when you're switching apps between, uh, uh, the browser or back to your Codex, you might be like, you know, doing some research. You might be writing a Google Doc. And you kind of hierarchically build up that understanding, and that plus, you know, existing connectors can give you a very good understanding of what you're trying to do. Um, I think people are very underestimating, like, how much the computer already knows.
- MCMartin Casado
Yeah.
- KLKen Liu
It's just there sitting there, you know, waiting to be tapped, so to speak.
- 5:59 – 8:16
Proactive AI: Reactive vs. Reflective
- JGJoel de la Garza
One, one of the interesting, having been a, a, a, an, also a very early adopter of the product, um, lately it's been just indispensable because it's able to catch It's sending me, um, messages now and letting me know that, hey, you've, you've got this conversation going on over here with this person, and it looks like the situation has changed. So we were planning an event yesterday. There was a 20% chance of weather, and in real time it was continually updating the forecast for us to let us know if we needed to get a tent or not, um, which is just, like, incredibly powerful. Like-
- MCMartin Casado
Did you ask it to do that?
- JGJoel de la Garza
No, I did not ask it to do that.
- MCMartin Casado
Yeah, yeah, yeah.
- JGJoel de la Garza
It like-
- MCMartin Casado
This is my exper- this is my experience with my Second, too.
- JGJoel de la Garza
Yeah.
- MCMartin Casado
Like, it literally, like, I'll get a text from my Second, and it's, like, something that I actually need to know about-
- JGJoel de la Garza
Yeah
- MCMartin Casado
... relative to, like, some ongoing conversation that I didn't even think about that, like, it actually is very thoughtful.
- JGJoel de la Garza
Yeah. And so, like, it's just, it's amazing. Like, I just... I'm genuinely curious, like, how are you executing on that? Like, things that I would've forgotten, it's literally catching.
- KLKen Liu
Yeah, yeah. So this goes back to the model of the person understanding what's, you know, might be important to them and what comes up at, at the, at this point. So I think proactivity is something that people talk a lot about, like pinging you, um, AI assisting pinging you when, um, at the time of need. But there are many levels of proactivity. Um, I would start, like, for example, the first level, which is called reactive ones. These are, this means, like, there's an email came in, and then there's a meme, right, online that's, [chuckles] you now get 10 notifications for email because, um, there are nine other personal agents pinging you about that. Um, reactive proactivity is, like, straightforward, easy to do, but what's harder is what I call reflective ones, which are things that connect the dots over, like, uh, many events and long time horizon, right? Stuff that realize, okay, months ago you said this, and recently you're working this towards your goal. This is your priority. These are some stuff on your to-do list, and by the way, you prefer this. And so kind of like connecting the dots will be able to give you much better proactivity, and I think that's kind of hints at, like, how maybe in the future personal, like, interactions with computers will go. Like, it, it would not be focused on reacting to things, but more like, okay, what are some, you know, interventions for the user that I can apply to? What are some things that I can, um, undo for the user without any asking? And I think that's one of the biggest bottlenecks right now. Like, people have to ask. You're not supposed to ask these things. They're super intelligence.
- JGJoel de la Garza
Yeah. I mean, you've built something that just kinda knows what I need at the moment. That's
- 8:16 – 10:27
Why You Only Need One Second
- JGJoel de la Garza
pretty impressive.
- MCMartin Casado
So I, I just, I just wanna make it very concrete for the people that are listening on what First P- Person Labs provides, um, and then I've got a question. So, so very concretely, um, it creates this thing called a Second. So you create something called a Second. Second has access to all of your digital contacts. I mean, it, it's amazing. It's like key strikes, uh, all your messages, all your channels, all your data. It does this in a very secure way, and based on that, it becomes basically your kinda personal butler, like that Alfred, right?
- KLKen Liu
Yeah.
- MCMartin Casado
And so, and then that interacts with your agents.
- KLKen Liu
Yeah.
- MCMartin Casado
So it's not an agent by itself. So this isn't like User Instinct or-
- KLKen Liu
Yeah
- MCMartin Casado
... GrokBot. It's its own... Or Dot. It's its own thing.
- KLKen Liu
Yeah.
- MCMartin Casado
Um, and then it provides that to the agents, right? So that's what it is. So here's an interesting thing. So for me, um, so I, I use GrokBot. I'm a big GrokBot user, and I have a bunch of them, right? I have one that keeps track of, like, my synthesizers, one that keeps track of my real estate, one that, you know, God forbid, helps me with my family, which I probably shouldn't need one, but I do. You know, so I just kind of have a bunch of these. Um, and then I have one Second, and they, th-they will interface with the Second. Do you imagine over time you'll have a second and a third and a fourth and a fifth? [laughs] Or do you think that it just, uh, you know, ends up being your one butler?
- KLKen Liu
Hmm. I think right now the, the fact that we have many personal agents is probably transitory. Like, agents will start to specialize just like people do. Um, some agents focus on one thing. Some f- agents focus on, let's say for example, browsing Amazon and shopping.
- MCMartin Casado
Yeah.
- KLKen Liu
They're extremely good at it because they have the data, they have the connections.
- MCMartin Casado
Yeah.
- KLKen Liu
And some agents will specifically focus on protecting your data-
- MCMartin Casado
Yeah
- KLKen Liu
... and aligning with your interests.
- MCMartin Casado
Yeah.
- KLKen Liu
Um, I think that's what Second is intended to be. Like, this is something that you can, should be able to trust to have your life's context in it.
- MCMartin Casado
Yeah.
- KLKen Liu
It's like a vault. And then over time, you know, this thing will, like, guide you, with you, like, you know, interface with the kind of the software world, so to speak, or even the hardware world.
- MCMartin Casado
So you think it'll stay as a Second?
- KLKen Liu
Yeah, it will stay as a Second. It will not be a third.
- MCMartin Casado
There will not be a third, there will not be a fourth.
- KLKen Liu
There will not be a third, fourth, yeah.
- MCMartin Casado
So, like, your one trusted agent for all your personal data that actually knows who you are, that will interact with all your other agents, that's Second.
- KLKen Liu
Yeah.
- JGJoel de la Garza
It's Batman's safe.
- KLKen Liu
Yeah.
- JGJoel de la Garza
[laughs]
- 10:27 – 15:41
Can You Trust Something That Sees Everything?
- JGJoel de la Garza
So I, I, I am, I am still a paranoid security person. Um, and, uh, I have been probably one of the faster adopters of, of some of this stuff, uh, compared to my-
- MCMartin Casado
Just to be clear, Joel took forever to be on social media. [laughs]
- JGJoel de la Garza
I just got on Twitter. [laughs]
- MCMartin Casado
It was like, it was like a few weeks ago, man. [laughs]
- JGJoel de la Garza
[laughs]
- MCMartin Casado
You guys-
- JGJoel de la Garza
I may be-
- MCMartin Casado
You guys have been in tech for 30 years.
- JGJoel de la Garza
I've been-
- MCMartin Casado
Just to be clear. [laughs]
- JGJoel de la Garza
Yeah, I'm a little paranoid. Um, [laughs] but I love Second, and, like, I, I gotta admit, like, just your background and kind of the things that are motivating you, and then also the long discussions we had about security architecture really got me comfortable. I mean, it'd be great to maybe just, just double-click, 'cause I, I think, I think when people think about this thing sees everything you do, there's an immediate sort of like, "Whoa," right?
- KLKen Liu
Yeah. Yeah, you're not supposed to trust it in the beginning.
- JGJoel de la Garza
Um, you're not supposed to trust it.
- KLKen Liu
Yeah.
- JGJoel de la Garza
And, and I think you've done a really excellent job of building something we can trust, and maybe just start to talk about, like, what that architecture is and kinda what's motivating you to build it.
- KLKen Liu
Yeah, yeah. So I think this is, to some extent, like, kind of like a n- like a n- a new form of, like, AI that is gonna work very differently from before. Like, before you initiate interaction, you get some results back. This time it's, like, more passive. It's way more ambient. It's gonna be there all the time. So, um, you know, you have to, like, kind of think about the way you, you know, tell people a bit about it, uh, like the way you architect it, like where the data is going. So at the end of the day, you know, there's only two things when it comes to AI. There's data, there's inference, right? So what you wanna do is, like, you have this data. You wanna define what inference you can bring to that data. Um, this is something kind of we built into Second. So first of all, you can get to choose what inference you can build. So if you don't trust, let's say, um, us In principle, you should be able to bring your own tokens.
- MCMartin Casado
Mm-hmm.
- KLKen Liu
And then just have those tokens power Second to then analyze data that's on your laptop locally, right?
- MCMartin Casado
But you also provide Second with an inference provider that uses trusted execution as well.
- KLKen Liu
Yes, yes. So over time, we imagine people not to bring their own tokens, because most people, when they bring their tokens, it's tokens from OpenAI or Anthropic.
- MCMartin Casado
Mm-hmm.
- KLKen Liu
Um, uh, you know, you, you wanna provide a stable option where you say, "If you use these tokens, then okay, no one's looking at the data," sometimes by policy guarantees, sometimes by hardware guarantees, right? So this, this kind of idea called confidential inference is getting, like, traction. It's the idea that you put a model inside some GPUs, and then the GPU, there's this special toggle, it's called a confidential computing, you can turn it on. And when you turn it on, you can basically prove that, um, uh, there's only the model running inside, no one can see inside. Um, and then you don't have to trust whoever's running that model. Um, the hardware is what you trust-
- MCMartin Casado
Yeah
- KLKen Liu
... which is NVIDIA. Um-
- MCMartin Casado
And so there, there's also another concept that, that's getting thrown around a lot right now, which is this concept of zero data retention, and that's very different from what you're describing.
- KLKen Liu
Yes, yes. Zero data retention is a promise by policy.
- MCMartin Casado
Mm-hmm.
- KLKen Liu
That means I'm a provider, I think I have to-
- MCMartin Casado
Basically, I give you my word.
- KLKen Liu
I give you my word.
- 15:41 – 17:40
Can an LLM Keep Your Secrets?
- KLKen Liu
Yeah.
- MCMartin Casado
How does, how does the... How does your Second know not to default information, though? Like, if I talk to your Second and I'm like, "Hey, Ken asked me to get his schedule." [laughs]
- KLKen Liu
[laughs] I asked mine the other day, "What's Joel hiding?"
- MCMartin Casado
[laughs] I know, right? You know, like, h- I mean, it just feels like when it comes to LLMs, it's hard to provide strong... Like, I get the systems aspect-
- KLKen Liu
Yeah, yeah
- MCMartin Casado
... it makes sense, but-
- KLKen Liu
Yeah
- MCMartin Casado
... seems hard to provide guarantees for-
- KLKen Liu
Yeah
- MCMartin Casado
... for the actual LLMs themselves.
- KLKen Liu
Yeah, yeah. So there's, uh, the deterministic guarantees, and then there's the, um, you know, LLM reasoning based guarantees.
- MCMartin Casado
Yeah, right.
- KLKen Liu
Yeah, the determinism based guarantees, you always have to implement them.
- MCMartin Casado
Gotcha. Yeah.
- KLKen Liu
Like, their apps usually, like-
- MCMartin Casado
Yeah
- KLKen Liu
... should not ever in the, retain in the system-
- MCMartin Casado
Yeah, yeah
- KLKen Liu
... passwords and all that. Now, with the reasoning based guarantees, you essentially, like, are falling back to, like, um, how much context do you have and under- using that context to then guide your decisions on what data to share.
- MCMartin Casado
Will I ever have access to your Second?
- KLKen Liu
Uh, no. Well, I mean, at least in a way that I would like, uh, allow it, right? So for example-
- MCMartin Casado
Oh, right
- KLKen Liu
... if you talk to my Second-
- MCMartin Casado
Yeah
- KLKen Liu
... um, for, I would have a model of Martin in it.
- MCMartin Casado
Yeah.
- KLKen Liu
It understands that these are interactions that we have.
- MCMartin Casado
Right.
- KLKen Liu
He does not send me a message at 3:00 a.m. And so if someone, if you send a message at 3:00 a.m., it's likely spam. You're likely, like-
- MCMartin Casado
Yeah
- 17:40 – 20:20
Cost, Models, and the Target User
- MCMartin Casado
Yeah. What models do you use? A bunch or are you into a certain model?
- KLKen Liu
A ton of models. Um-
- MCMartin Casado
Yeah
- KLKen Liu
... actually, this, these two, another point we can talk about, which is the idea of portability.
- MCMartin Casado
Yeah, yeah.
- KLKen Liu
You can run this on any models, and you should be able to run this on any models because later on you want to run it on local inference.
- MCMartin Casado
Right.
- KLKen Liu
Um, but yeah, like right now, you, there, there are definitely a lot of things that are capability bound. You wanna make sure that you use the smart models for it. Um, but ultimately models are pretty smart these days.
- MCMartin Casado
It also feels kind of, like, heavyweight and expensive. Like, there's a lot of data [laughs] and like whatever. So can you talk about, like, the target user? Is it just, like, any person? Is it a professional? Is it- The Joels of the world.
- KLKen Liu
[laughs] Um, initially something like this, like something that looks at your traces 24/7 and analyze them means, like, running a lot of tokens, right? Uh, let's... We can do some quick math here. Let's say you have, uh, one agent, or actually one agent running 24/7, and let's say at 60 tokens per second, and let's say you run 10 of those, uh, because there's so much to analyze. And at Opus 5i's pricing, that leads to $1,000 per day.
- MCMartin Casado
Oh my goodness, $1,000 per day? Yeah.
- KLKen Liu
Yeah, for Opus. Oh, if you, if you use Deep- if you use DeepSeek Flash, for example, it becomes like $60 per day.
- MCMartin Casado
Yeah.
- KLKen Liu
Right? And now this cost is gonna go down very quickly. Um, right now it's not gonna be for everyone.
- MCMartin Casado
Yeah.
- KLKen Liu
Um, uh, especially, like, not everyone has this problem of I need to understand my context all the time.
- MCMartin Casado
Yeah, yeah.
- KLKen Liu
Like you maybe be busy professionals like yourself.
- MCMartin Casado
Yeah.
- KLKen Liu
And, um, uh, people wanna switch, like, between tools very, very frequently.
- MCMartin Casado
Right. Yeah.
- KLKen Liu
Um, but over time, I imagine, like, just, you know, since we started this, this company actually, uh, token prices already dropped like by more than 50%. Like, just new models come out.
- MCMartin Casado
Yeah, yeah, yeah.
- KLKen Liu
They're more capable.
- MCMartin Casado
Yeah.
- KLKen Liu
You're swapping them in.
- MCMartin Casado
Yeah.
- KLKen Liu
Then, yeah, that you just keep going down.
- MCMartin Casado
Is there, is there a way to, like, just do the, like, not super professional version where you actually reduce the amount of tokens? Like kind of a de- not degraded, but like kind of like a, like a looser SLA. Is that a thing, or...?
- KLKen Liu
You can... You, you in principle, you should be able to do that.
- 20:20 – 23:34
Use Cases and the Model Mix
- MCMartin Casado
When is, um, when is this available?
- KLKen Liu
Uh, it's available today.
- MCMartin Casado
Yeah.
- KLKen Liu
But right now the problem is that, uh, we have to scale up the, uh, the inference capacity a little bit more and support. Uh, we wanna make sure this rolls out in, in a very, like, responsible way. I wanna make sure people's data, like, are controlled and secure and because this is a, a new kind of a tool that, you know, really touches on a lot of data.
- MCMartin Casado
It'd be actually interesting 'cause I, you know, again, as a user, I found it's, it's surprisingly useful in ways I wouldn't have anticipated. You're probably like the top power user, right?
- KLKen Liu
Yeah.
- MCMartin Casado
Because you created it and [laughs] -
- KLKen Liu
Mm.
- MCMartin Casado
So maybe can you talk to what you think are, like, the more surprising use cases that you found, like kind of more useful and surprising use cases?
- KLKen Liu
For us, for development, it turns out that, uh, it's just a super, super good memory of what we're trying to do. Um, you no longer have to explain anything to your coding agents. You can just screenshot anything-
- MCMartin Casado
Wow
- KLKen Liu
... and just say, "Fix."
- MCMartin Casado
Yeah.
- KLKen Liu
And then you would know what you're fixing, what you're trying to fix. And, uh, I also am on the go a lot of the times, so what you can do is like, um, something I call context continuity. It's that like, you know, you, you, you browse something on the laptop, you close it, and then you take out your phone. It's the same context. It just knows you up to the minute, and you can continue doing stuff there. You can write that email, um, with you. And I think we are at the stage where this is still pretty, pretty early in understanding what the use cases are. Like, we are really underserving the amount of like context that the system has. Like, there's a lot of like digging and application building and interaction design-
- MCMartin Casado
Yeah
- KLKen Liu
... to like, like, kind of like unleash the, the potential of this.
- JGJoel de la Garza
And it seems like with these things, the more context you get, the better the models work, right?
- KLKen Liu
Yeah.
- JGJoel de la Garza
That's been the, the opening. I'm curious 'cause we were talking about model... switching between models and model portability and model cost, and like I gotta... I mean, I'm just curious, like I, I haven't seen a whole lot of difference between using like a, a world-class frontier model versus using the more commodity ones. I'm curious, have you guys seen that for, for, for doing a lot of this stuff with Second?
- KLKen Liu
Yeah, this is a good question. So if, if you say like, if you say, "I wanna analyze what is a semantically meaningful moment that happened on your computer." Let's say you go to a calendar and you move the invite. That you don't need, um, Fable.
- JGJoel de la Garza
Yeah. [laughs]
- KLKen Liu
Um, uh, there are many tasks like that. Essentially, it's like a hierarchy of information you wanna understand. Um, a different kind of information would be like, "What is Joel's goal for this week?"
- JGJoel de la Garza
Mm-hmm.
- KLKen Liu
That's something you want a very expensive model, but you don't want, you don't need to run it very frequently.
- JGJoel de la Garza
Mm-hmm.
- KLKen Liu
So essentially, Second involves like a harness, it's like a harness design problem, like what, what models gets used when-
- JGJoel de la Garza
Mm-hmm
- KLKen Liu
... and so on. Um, and so that's where you have to build essentially this system to be like portable from the beginning. There are always gonna be more models come out. Like Jeff, for example, came out. It turns out that Jeff is s- extremely good at retrieval, right?
- MCMartin Casado
Really?
- KLKen Liu
Yeah.
- 23:34 – 26:20
How You Use Second (APIs and Apps)
- MCMartin Casado
So, so maybe let's say es- uh, let's assume like someone's, um, you know, listening, like starts to use Second or kind of signs up. Like maybe talk a little bit about like the product, what they have access to, like, you know, something a little bit more concretely about like how, how, how a human interfaces with this-
- KLKen Liu
Mm
- MCMartin Casado
... what controls they have, et cetera.
- KLKen Liu
Yeah. So today the way it works is that you install an app.
- MCMartin Casado
Okay.
- KLKen Liu
The app asks for some permissions.
- MCMartin Casado
Right.
- KLKen Liu
Um, uh, and then you give the permissions, and then you specify what your tokens are, and then it just runs a bunch of... It starts running agents on top.
- MCMartin Casado
Yeah.
- KLKen Liu
Uh, not-
- MCMartin Casado
Yeah, I, I remember when I in- installed, I was like, you know, connect to Slack, connect to Signal, connect to WhatsApp.
- KLKen Liu
Yeah.
- MCMartin Casado
You know, is that what you're talking about?
- KLKen Liu
Uh, not, not, not that. Uh-
- MCMartin Casado
Oh.
- KLKen Liu
I mean, that's part of it.
- MCMartin Casado
Yeah.
- KLKen Liu
But also the, the raw computer stream essentially.
- MCMartin Casado
Oh, right. Yeah, yeah, yeah.
- KLKen Liu
Yeah. So, and, and the connectors help, you know. It's, it's like a sample efficiency problem. It makes it learn faster.
- MCMartin Casado
I see. I see.
- KLKen Liu
Yeah. Um- Uh, s- and, and it starts running background agents, and there's pretty much nothing you gotta do from there-
- MCMartin Casado
Yeah
- KLKen Liu
... because the background agents keep, keep running, and then you start producing these, like, APIs out of, out of you, right? Like, there's a API about, uh, uh, uh, what, what you just did 10 minutes ago.
- MCMartin Casado
Yeah.
- KLKen Liu
And there's a API for what are you gonna do, uh, for the next, next week, what are your priorities.
- MCMartin Casado
Yeah.
- KLKen Liu
There's a API for your reminders, and there's a API for your wiki, like-
- MCMartin Casado
Yeah
- KLKen Liu
... people in your life, processes in your life.
- 26:20 – 30:43
Toward Jarvis: AI Everywhere
- MCMartin Casado
Are you planning to evolve the, the interface? I mean, you've just spent a lot of time talking about how it may evolve, but I always view Second as, like, out of view of an agent where you talk to it, you know, and, like, the UI is somebody else's problem. But are you also, like, actually anticipating innovating there?
- KLKen Liu
Um, I think it will be something that we kind of focus on in the, in the long run.
- MCMartin Casado
Yeah.
- KLKen Liu
Um, because, like, uh, right now it's a systems problem and, uh, you know, the understanding problem comes first.
- MCMartin Casado
Can I, can I just put in a request?
- KLKen Liu
Mm-hmm.
- MCMartin Casado
Okay, so I want, I want to go to my house-
- KLKen Liu
Yeah
- MCMartin Casado
... and I want, like, that 1970s AI, [laughs] you know, like, the head on the screen that I can talk to [laughs] and, like...
- KLKen Liu
Yeah.
- MCMartin Casado
You know, it's surprising. We have these agents, and they're so smart. Like, my Second is so smart. I, I just... Wow. And I'll ask it questions. It's very good, but it's not embodied, and it drives me nuts, man. [laughs]
- KLKen Liu
[laughs]
- MCMartin Casado
So like... [laughs]
- KLKen Liu
Yeah.
- MCMartin Casado
If in the future you can have Alfred actually be an Alfred, I'll be very happy.
- KLKen Liu
Yeah, yeah. I, I think we, we see a lot of this in, in the movies, right? Like, there's Jarvis. People talk to Jarvis all the time.
- MCMartin Casado
But why, and why, why have not...
- KLKen Liu
We don't have Jarvis, and there's many aspects of Jarvis that people underlook. It's not just that it's a, it's intelligence and unders- like, it can do things.
- MCMartin Casado
Yeah.
- KLKen Liu
But it's more that it's everywhere. Like, it's like-
- MCMartin Casado
Right
- KLKen Liu
... it's on your door. It's in your armor. It's, it's-
- MCMartin Casado
Yeah
- KLKen Liu
... you know, in, in your screen, and that kind of con- continuity and ubiquity, it's something, it's hard to-
- MCMartin Casado
I, I, I'll say, I'll say, like, like, kind of my, my, my visions of the sci-fi future basically started when I started using Second. I'm like, finally there's something that, like, feels like it's got context. It feels like it knows me. Like, I can talk about it. It feels like it's a relevant conversation, and so I do feel like we're a bit ste- you know, a step closer to that. But, you know, I mean, we're, we're still tied to these devices, so-
- KLKen Liu
Yeah
- MCMartin Casado
... you know, I don't know if it's gonna be in a robot or it's gonna be, like, in a screen or, or whatever it is, but it would be actually nice to have this thing somewhat disconnected from my phone and my...
- KLKen Liu
Yeah.
- MCMartin Casado
Yeah.
- KLKen Liu
Yeah. In the long future, I imagine that every person will have... Like, essentially everyone will has, have a Second, and all the devices just becomes inputs.
- 30:43 – 32:56
Fort Knox Meets Delightful UX
- KLKen Liu
on-
- MCMartin Casado
So I j- I just have one more question, which is, um- Uh, and, and, and, and Joel, Joel kind of touched on this. Like, you're like such a hardcore systems guy. Um, like, I mean, you know, you work... You know, who's your advisor, Percy?
- KLKen Liu
Percy Liang, yes, at Andreessen.
- MCMartin Casado
He also worked with Jamon A., hardcore, you know-
- KLKen Liu
Yeah
- MCMartin Casado
... um, security guy. Um, and you know, a lot of the, the, the path that took you here was this kind of hardcore security stuff, and so you're building this pretty serious system. You know, it's very secure, um, and very functional. On the other hand, when I interface with it, it's like this lovely user experience, and so like, is there a bit of a schizophrenia in the company where you both need to be like this super hardcore, you know, build the Fort Knox infrastructure, but also make it very usable? Or does like the LLM basically solve, you know, the, the interface problem?
- KLKen Liu
Yeah, yeah. There, there are two kinds of problem we're dealing with, the infrastructure problem and the interaction problem.
- MCMartin Casado
Yeah.
- KLKen Liu
And we've made our mistakes- [laughs]
- MCMartin Casado
These are mostly, but they're totally different. [laughs]
- KLKen Liu
Like, very.
- MCMartin Casado
Like you're as low on the stack as it gets.
- KLKen Liu
Yeah.
- MCMartin Casado
I mean, that's not hardware.
- KLKen Liu
Yeah. So we, we, we made a lot of mistakes on the interaction side. We tried to go all in on interaction, and turns out it's actually very hard. There's a lot of details that go in-
- MCMartin Casado
Yeah
- KLKen Liu
... you know, to create that delightful experience.
- MCMartin Casado
Oh, you mean like actually creating a UI?
- KLKen Liu
Yeah.
- MCMartin Casado
Yeah, that's right.
- KLKen Liu
UI, but also like, you know... Yeah, yeah, like what kind of like personal pages you can create on top of this. I mean, and we realized that, you know, like it's, it's something that would take a long time to get right, and this is a time like everyone would take this kind of like understanding of themselves and build this. And so we want to enable people to do that instead of doing it ourselves. And, um, so we decided to like close down the surfaces that we have. There's only texting, there's only API, and there's nothing else.
- MCMartin Casado
Right.
- KLKen Liu
You can technically use it as a full agent, like a co-work agent on, on your, on your laptop, like it runs Bash and all that.
- MCMartin Casado
Yeah.
- KLKen Liu
Um, but we meant it as like very minimal-
- MCMartin Casado
Got it
- KLKen Liu
... so we can like make the interaction like more, more smooth.
- MCMartin Casado
Ken, thanks for coming in. Um, congratulations on everything, uh, and the launch. And is there anything that you want to, uh, leave the listeners with?
- KLKen Liu
Go try it out today. Um, it's available, uh, it's on 2.computer. Uh, it's a very short URL, and we hope to, like give you access as soon as possible. And if you resonate with this, um, mission, you know, come join us. There's a lot to build.
- MCMartin Casado
Great. Thank you so much.
Episode duration: 33:10
In the uListen app, the transcript lights up word by word as you listen. Get uListen
Transcript of episode kBJe4H_WobA