Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

a16za16z

What It Takes to Build an AI That Actually Knows You

AI agents are getting smarter, but every new agent still has to learn who you are. What if your preferences, priorities, relationships, and digital context could move with you instead?

Ken LiuguestMartin CasadohostJoel de la Garzahost
Nov 18, 202633mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

  1. 0:00 – 0:50

    Intro

    1. KL

      People are very underestimating how much the computer already knows. You no longer have to explain anything to your coding agents. You can just screenshot anything and just say, "Fix," and then you would know what you're fixing. The question then is like, well, can you just like analyze the heck out of it?

    2. MC

      What context does Grok Bot not have that First Person Labs would allow?

    3. KL

      One of the biggest problems right now, like people have to ask. You're not supposed to ask these things. They're super intelligence. But if you have a user model that just builds up by itself, and then these things just keep analyzing the background, then you can just share that over instantly. We focus on essentially creating a model of a person and then put that person in control.

    4. JG

      When people think about this thing sees everything you do, there's an immediate sort of like, whoa.

    5. KL

      If you don't trust, let's say, us, in principle, you should be able to bring your own tokens to then analyze data that's on your laptop locally.

    6. MC

      Can you talk through what you think are like the more surprising use cases that you found?

    7. KL

      For development, it turns out that-

  2. 0:50 – 2:43

    What Is First Person Labs?

    1. JG

      Thank you, Ken, for joining us. Really happy to have you on to talk about, uh, what you've been working on at First Person Labs.

    2. MC

      Is this the first time you're talking about it?

    3. KL

      Yes.

    4. MC

      Yes. So what do you do?

    5. KL

      So we focus on essentially creating a model of a person and then put that person in control. And the model of a person basically means that understanding their memories, understanding what they do, but more of that, uh, their priorities, preferences, goals, values, and commitments, and intent.

    6. MC

      So as-- Sorry, just let's just make this very simple. I'm a big Grok Bot user, so I use Grok Bot a lot. Where would First Person Labs fit into my usage of Grok Bot?

    7. KL

      Yeah. The way, the way it works is that you would just bring a context over to Grok Bot. Um, it's like portable context. You go to Grok Bot and you say, "Hey, can you talk to my thing to figure out what I want?" And then the two of them will start talking, and then you can just send prompts like, oh, buy me a flight, and you would know what flight you're talking about.

    8. MC

      What, what context does Grok Bot not have that First Person Labs would allow?

    9. KL

      I would ask-- I would probably ask the opposite question, like what, what, what does First Person Labs not have if you build a user model for yourself, right? Let's say your preferences on the seat on your airplane, the preferences of your, uh, uh, how early you would like to arrive at the airport, and, um, like where are you flying? Uh, are you flying for a class or, or, or normal? And all of that, right, as of today, you have to explain that to Grok Bot. You have like, okay, there's multiple rounds of like questioning. But if you have a user model or like a model view that just builds up by itself, and then these things just keep analyzing the background, then you can just sh-share that over, uh, instantly.

    10. JG

      So for example, the OpenAI released two new cyber-capable models yesterday.

    11. KL

      Mm.

    12. JG

      And I haven't been using my Codex in a while, but I have been using Second, so I can just go start using Codex and it has all the other work that I've already done and all the additional context already saved?

    13. KL

      Yeah. Yeah. Um, you just go to Codex and like, "How's my Second? What's going on?"

    14. JG

      That's awesome.

  3. 2:43 – 3:56

    The Alfred Model: Your Agent's Butler

    1. MC

      Does Second ever be a first person? [laughs] Will it ever be a first or will it always go through agents?

    2. KL

      Uh, I, that's a good question. I think user model is a, it's a, it's a, it's a process. You have to start somewhere. You start with a baseline foundational list of artifacts, what you did, uh, who are the people in your life, maybe your preferences. And over time, it starts analyzing like the way you talk-

    3. MC

      Yeah

    4. KL

      ... the way you, uh, present yourself, the way you share information.

    5. MC

      Yeah.

    6. KL

      And over time, you, you would like build up even more artifacts on top and more understanding on top, and maybe some of that you train into the model. And so, um, over time, you will go from a portable context into something more of a representation of yourself. Um, one example I like to think of is this like, um, Alfred and Batman example, where if, let's say, Batman wants to buy, buy, buy a cheeseburger and then get Alfred to do it, um, Alfred would not reveal, for example, that it's Batman doing it, but he would just buy it. He would state the preference.

    7. MC

      Alfred being Batman's butler.

    8. KL

      Oh, yes, yes, yes.

    9. MC

      Okay.

    10. KL

      Being the butler, yeah.

    11. JG

      Or his agent.

    12. KL

      Yeah, yeah. Or his agent.

    13. JG

      Or his agent. [laughs]

    14. KL

      Yeah, yeah.

    15. JG

      I should call my agent Alfred.

    16. MC

      That's a great idea.

    17. KL

      Yeah, yeah. He would just do it. Um, he would just do it, and then, uh, you, you would trust him because you know that they have a good model of you and, uh, what that means in, when it comes to getting things done.

  4. 3:56 – 5:59

    What Your Computer Already Knows About You

    1. MC

      It's, it's inter- it's interesting, which is like, so you de- you describe First Person, Person Lab as, as having this thing called a Second. The Second understands who you are and has context, and then other agents talk to that Second, right?

    2. KL

      Yeah.

    3. MC

      And so, like, it kind of understands your preferences, it understands what you're doing, what you wanna do. Um, but I, I think that that actually doesn't highlight another thing that you've been working very hard on, and I think it's worth talking about, is like you're actually able to get access to context, um, digital context that a person has, like messages, chat, kind of email in a way, and make that also available. So can you talk a little bit about that? Like, so here, being a, one of the earliest First Person Lab users-

    4. KL

      Yeah

    5. MC

      ... I would say the thing that struck me was that, not that it knew who I was, but more it had access to all of my channels very well. So maybe... And then by the way, which it turns out to be a very hard systems problem, so maybe talk a little bit about that aspect of what you're doing.

    6. KL

      Yeah. Um, it is indeed a very hard problem, especially to get it secured private. Um, so the way it roughly works is that right now on your computer, um, there's like an infinite, an infinite stream of events. As you use your computer, you know, the apps that you switch to, the way you pause on certain websites, the way you type, and the way, like, your Wi-Fi is connected, and that the OS already gives you. Um, this is across Apple, Microsoft, and even Linux. Um, and the question then is like, well, can you just like analyze the heck out, the heck out of it, right? Like this raw stream twenty-four/seven, apply agents on top, twenty-four/seven agents. So I can first synthesize, for example, well, this piece of typing at this particular app might mean that you're sending a message. And then when you're switching apps between, uh, uh, the browser or back to your Codex, you might be like, you know, doing some research. You might be writing a Google Doc. And you kind of hierarchically build up that understanding, and that plus, you know, existing connectors can give you a very good understanding of what you're trying to do. Um, I think people are very underestimating, like, how much the computer already knows.

    7. MC

      Yeah.

    8. KL

      It's just there sitting there, you know, waiting to be tapped, so to speak.

  5. 5:59 – 8:16

    Proactive AI: Reactive vs. Reflective

    1. JG

      One, one of the interesting, having been a, a, a, an, also a very early adopter of the product, um, lately it's been just indispensable because it's able to catch It's sending me, um, messages now and letting me know that, hey, you've, you've got this conversation going on over here with this person, and it looks like the situation has changed. So we were planning an event yesterday. There was a 20% chance of weather, and in real time it was continually updating the forecast for us to let us know if we needed to get a tent or not, um, which is just, like, incredibly powerful. Like-

    2. MC

      Did you ask it to do that?

    3. JG

      No, I did not ask it to do that.

    4. MC

      Yeah, yeah, yeah.

    5. JG

      It like-

    6. MC

      This is my exper- this is my experience with my Second, too.

    7. JG

      Yeah.

    8. MC

      Like, it literally, like, I'll get a text from my Second, and it's, like, something that I actually need to know about-

    9. JG

      Yeah

    10. MC

      ... relative to, like, some ongoing conversation that I didn't even think about that, like, it actually is very thoughtful.

    11. JG

      Yeah. And so, like, it's just, it's amazing. Like, I just... I'm genuinely curious, like, how are you executing on that? Like, things that I would've forgotten, it's literally catching.

    12. KL

      Yeah, yeah. So this goes back to the model of the person understanding what's, you know, might be important to them and what comes up at, at the, at this point. So I think proactivity is something that people talk a lot about, like pinging you, um, AI assisting pinging you when, um, at the time of need. But there are many levels of proactivity. Um, I would start, like, for example, the first level, which is called reactive ones. These are, this means, like, there's an email came in, and then there's a meme, right, online that's, [chuckles] you now get 10 notifications for email because, um, there are nine other personal agents pinging you about that. Um, reactive proactivity is, like, straightforward, easy to do, but what's harder is what I call reflective ones, which are things that connect the dots over, like, uh, many events and long time horizon, right? Stuff that realize, okay, months ago you said this, and recently you're working this towards your goal. This is your priority. These are some stuff on your to-do list, and by the way, you prefer this. And so kind of like connecting the dots will be able to give you much better proactivity, and I think that's kind of hints at, like, how maybe in the future personal, like, interactions with computers will go. Like, it, it would not be focused on reacting to things, but more like, okay, what are some, you know, interventions for the user that I can apply to? What are some things that I can, um, undo for the user without any asking? And I think that's one of the biggest bottlenecks right now. Like, people have to ask. You're not supposed to ask these things. They're super intelligence.

    13. JG

      Yeah. I mean, you've built something that just kinda knows what I need at the moment. That's

  6. 8:16 – 10:27

    Why You Only Need One Second

    1. JG

      pretty impressive.

    2. MC

      So I, I just, I just wanna make it very concrete for the people that are listening on what First P- Person Labs provides, um, and then I've got a question. So, so very concretely, um, it creates this thing called a Second. So you create something called a Second. Second has access to all of your digital contacts. I mean, it, it's amazing. It's like key strikes, uh, all your messages, all your channels, all your data. It does this in a very secure way, and based on that, it becomes basically your kinda personal butler, like that Alfred, right?

    3. KL

      Yeah.

    4. MC

      And so, and then that interacts with your agents.

    5. KL

      Yeah.

    6. MC

      So it's not an agent by itself. So this isn't like User Instinct or-

    7. KL

      Yeah

    8. MC

      ... GrokBot. It's its own... Or Dot. It's its own thing.

    9. KL

      Yeah.

    10. MC

      Um, and then it provides that to the agents, right? So that's what it is. So here's an interesting thing. So for me, um, so I, I use GrokBot. I'm a big GrokBot user, and I have a bunch of them, right? I have one that keeps track of, like, my synthesizers, one that keeps track of my real estate, one that, you know, God forbid, helps me with my family, which I probably shouldn't need one, but I do. You know, so I just kind of have a bunch of these. Um, and then I have one Second, and they, th-they will interface with the Second. Do you imagine over time you'll have a second and a third and a fourth and a fifth? [laughs] Or do you think that it just, uh, you know, ends up being your one butler?

    11. KL

      Hmm. I think right now the, the fact that we have many personal agents is probably transitory. Like, agents will start to specialize just like people do. Um, some agents focus on one thing. Some f- agents focus on, let's say for example, browsing Amazon and shopping.

    12. MC

      Yeah.

    13. KL

      They're extremely good at it because they have the data, they have the connections.

    14. MC

      Yeah.

    15. KL

      And some agents will specifically focus on protecting your data-

    16. MC

      Yeah

    17. KL

      ... and aligning with your interests.

    18. MC

      Yeah.

    19. KL

      Um, I think that's what Second is intended to be. Like, this is something that you can, should be able to trust to have your life's context in it.

    20. MC

      Yeah.

    21. KL

      It's like a vault. And then over time, you know, this thing will, like, guide you, with you, like, you know, interface with the kind of the software world, so to speak, or even the hardware world.

    22. MC

      So you think it'll stay as a Second?

    23. KL

      Yeah, it will stay as a Second. It will not be a third.

    24. MC

      There will not be a third, there will not be a fourth.

    25. KL

      There will not be a third, fourth, yeah.

    26. MC

      So, like, your one trusted agent for all your personal data that actually knows who you are, that will interact with all your other agents, that's Second.

    27. KL

      Yeah.

    28. JG

      It's Batman's safe.

    29. KL

      Yeah.

    30. JG

      [laughs]

  7. 10:27 – 15:41

    Can You Trust Something That Sees Everything?

    1. JG

      So I, I, I am, I am still a paranoid security person. Um, and, uh, I have been probably one of the faster adopters of, of some of this stuff, uh, compared to my-

    2. MC

      Just to be clear, Joel took forever to be on social media. [laughs]

    3. JG

      I just got on Twitter. [laughs]

    4. MC

      It was like, it was like a few weeks ago, man. [laughs]

    5. JG

      [laughs]

    6. MC

      You guys-

    7. JG

      I may be-

    8. MC

      You guys have been in tech for 30 years.

    9. JG

      I've been-

    10. MC

      Just to be clear. [laughs]

    11. JG

      Yeah, I'm a little paranoid. Um, [laughs] but I love Second, and, like, I, I gotta admit, like, just your background and kind of the things that are motivating you, and then also the long discussions we had about security architecture really got me comfortable. I mean, it'd be great to maybe just, just double-click, 'cause I, I think, I think when people think about this thing sees everything you do, there's an immediate sort of like, "Whoa," right?

    12. KL

      Yeah. Yeah, you're not supposed to trust it in the beginning.

    13. JG

      Um, you're not supposed to trust it.

    14. KL

      Yeah.

    15. JG

      And, and I think you've done a really excellent job of building something we can trust, and maybe just start to talk about, like, what that architecture is and kinda what's motivating you to build it.

    16. KL

      Yeah, yeah. So I think this is, to some extent, like, kind of like a n- like a n- a new form of, like, AI that is gonna work very differently from before. Like, before you initiate interaction, you get some results back. This time it's, like, more passive. It's way more ambient. It's gonna be there all the time. So, um, you know, you have to, like, kind of think about the way you, you know, tell people a bit about it, uh, like the way you architect it, like where the data is going. So at the end of the day, you know, there's only two things when it comes to AI. There's data, there's inference, right? So what you wanna do is, like, you have this data. You wanna define what inference you can bring to that data. Um, this is something kind of we built into Second. So first of all, you can get to choose what inference you can build. So if you don't trust, let's say, um, us In principle, you should be able to bring your own tokens.

    17. MC

      Mm-hmm.

    18. KL

      And then just have those tokens power Second to then analyze data that's on your laptop locally, right?

    19. MC

      But you also provide Second with an inference provider that uses trusted execution as well.

    20. KL

      Yes, yes. So over time, we imagine people not to bring their own tokens, because most people, when they bring their tokens, it's tokens from OpenAI or Anthropic.

    21. MC

      Mm-hmm.

    22. KL

      Um, uh, you know, you, you wanna provide a stable option where you say, "If you use these tokens, then okay, no one's looking at the data," sometimes by policy guarantees, sometimes by hardware guarantees, right? So this, this kind of idea called confidential inference is getting, like, traction. It's the idea that you put a model inside some GPUs, and then the GPU, there's this special toggle, it's called a confidential computing, you can turn it on. And when you turn it on, you can basically prove that, um, uh, there's only the model running inside, no one can see inside. Um, and then you don't have to trust whoever's running that model. Um, the hardware is what you trust-

    23. MC

      Yeah

    24. KL

      ... which is NVIDIA. Um-

    25. MC

      And so there, there's also another concept that, that's getting thrown around a lot right now, which is this concept of zero data retention, and that's very different from what you're describing.

    26. KL

      Yes, yes. Zero data retention is a promise by policy.

    27. MC

      Mm-hmm.

    28. KL

      That means I'm a provider, I think I have to-

    29. MC

      Basically, I give you my word.

    30. KL

      I give you my word.

  8. 15:41 – 17:40

    Can an LLM Keep Your Secrets?

    1. KL

      Yeah.

    2. MC

      How does, how does the... How does your Second know not to default information, though? Like, if I talk to your Second and I'm like, "Hey, Ken asked me to get his schedule." [laughs]

    3. KL

      [laughs] I asked mine the other day, "What's Joel hiding?"

    4. MC

      [laughs] I know, right? You know, like, h- I mean, it just feels like when it comes to LLMs, it's hard to provide strong... Like, I get the systems aspect-

    5. KL

      Yeah, yeah

    6. MC

      ... it makes sense, but-

    7. KL

      Yeah

    8. MC

      ... seems hard to provide guarantees for-

    9. KL

      Yeah

    10. MC

      ... for the actual LLMs themselves.

    11. KL

      Yeah, yeah. So there's, uh, the deterministic guarantees, and then there's the, um, you know, LLM reasoning based guarantees.

    12. MC

      Yeah, right.

    13. KL

      Yeah, the determinism based guarantees, you always have to implement them.

    14. MC

      Gotcha. Yeah.

    15. KL

      Like, their apps usually, like-

    16. MC

      Yeah

    17. KL

      ... should not ever in the, retain in the system-

    18. MC

      Yeah, yeah

    19. KL

      ... passwords and all that. Now, with the reasoning based guarantees, you essentially, like, are falling back to, like, um, how much context do you have and under- using that context to then guide your decisions on what data to share.

    20. MC

      Will I ever have access to your Second?

    21. KL

      Uh, no. Well, I mean, at least in a way that I would like, uh, allow it, right? So for example-

    22. MC

      Oh, right

    23. KL

      ... if you talk to my Second-

    24. MC

      Yeah

    25. KL

      ... um, for, I would have a model of Martin in it.

    26. MC

      Yeah.

    27. KL

      It understands that these are interactions that we have.

    28. MC

      Right.

    29. KL

      He does not send me a message at 3:00 a.m. And so if someone, if you send a message at 3:00 a.m., it's likely spam. You're likely, like-

    30. MC

      Yeah

  9. 17:40 – 20:20

    Cost, Models, and the Target User

    1. MC

      Yeah. What models do you use? A bunch or are you into a certain model?

    2. KL

      A ton of models. Um-

    3. MC

      Yeah

    4. KL

      ... actually, this, these two, another point we can talk about, which is the idea of portability.

    5. MC

      Yeah, yeah.

    6. KL

      You can run this on any models, and you should be able to run this on any models because later on you want to run it on local inference.

    7. MC

      Right.

    8. KL

      Um, but yeah, like right now, you, there, there are definitely a lot of things that are capability bound. You wanna make sure that you use the smart models for it. Um, but ultimately models are pretty smart these days.

    9. MC

      It also feels kind of, like, heavyweight and expensive. Like, there's a lot of data [laughs] and like whatever. So can you talk about, like, the target user? Is it just, like, any person? Is it a professional? Is it- The Joels of the world.

    10. KL

      [laughs] Um, initially something like this, like something that looks at your traces 24/7 and analyze them means, like, running a lot of tokens, right? Uh, let's... We can do some quick math here. Let's say you have, uh, one agent, or actually one agent running 24/7, and let's say at 60 tokens per second, and let's say you run 10 of those, uh, because there's so much to analyze. And at Opus 5i's pricing, that leads to $1,000 per day.

    11. MC

      Oh my goodness, $1,000 per day? Yeah.

    12. KL

      Yeah, for Opus. Oh, if you, if you use Deep- if you use DeepSeek Flash, for example, it becomes like $60 per day.

    13. MC

      Yeah.

    14. KL

      Right? And now this cost is gonna go down very quickly. Um, right now it's not gonna be for everyone.

    15. MC

      Yeah.

    16. KL

      Um, uh, especially, like, not everyone has this problem of I need to understand my context all the time.

    17. MC

      Yeah, yeah.

    18. KL

      Like you maybe be busy professionals like yourself.

    19. MC

      Yeah.

    20. KL

      And, um, uh, people wanna switch, like, between tools very, very frequently.

    21. MC

      Right. Yeah.

    22. KL

      Um, but over time, I imagine, like, just, you know, since we started this, this company actually, uh, token prices already dropped like by more than 50%. Like, just new models come out.

    23. MC

      Yeah, yeah, yeah.

    24. KL

      They're more capable.

    25. MC

      Yeah.

    26. KL

      You're swapping them in.

    27. MC

      Yeah.

    28. KL

      Then, yeah, that you just keep going down.

    29. MC

      Is there, is there a way to, like, just do the, like, not super professional version where you actually reduce the amount of tokens? Like kind of a de- not degraded, but like kind of like a, like a looser SLA. Is that a thing, or...?

    30. KL

      You can... You, you in principle, you should be able to do that.

  10. 20:20 – 23:34

    Use Cases and the Model Mix

    1. MC

      When is, um, when is this available?

    2. KL

      Uh, it's available today.

    3. MC

      Yeah.

    4. KL

      But right now the problem is that, uh, we have to scale up the, uh, the inference capacity a little bit more and support. Uh, we wanna make sure this rolls out in, in a very, like, responsible way. I wanna make sure people's data, like, are controlled and secure and because this is a, a new kind of a tool that, you know, really touches on a lot of data.

    5. MC

      It'd be actually interesting 'cause I, you know, again, as a user, I found it's, it's surprisingly useful in ways I wouldn't have anticipated. You're probably like the top power user, right?

    6. KL

      Yeah.

    7. MC

      Because you created it and [laughs] -

    8. KL

      Mm.

    9. MC

      So maybe can you talk to what you think are, like, the more surprising use cases that you found, like kind of more useful and surprising use cases?

    10. KL

      For us, for development, it turns out that, uh, it's just a super, super good memory of what we're trying to do. Um, you no longer have to explain anything to your coding agents. You can just screenshot anything-

    11. MC

      Wow

    12. KL

      ... and just say, "Fix."

    13. MC

      Yeah.

    14. KL

      And then you would know what you're fixing, what you're trying to fix. And, uh, I also am on the go a lot of the times, so what you can do is like, um, something I call context continuity. It's that like, you know, you, you, you browse something on the laptop, you close it, and then you take out your phone. It's the same context. It just knows you up to the minute, and you can continue doing stuff there. You can write that email, um, with you. And I think we are at the stage where this is still pretty, pretty early in understanding what the use cases are. Like, we are really underserving the amount of like context that the system has. Like, there's a lot of like digging and application building and interaction design-

    15. MC

      Yeah

    16. KL

      ... to like, like, kind of like unleash the, the potential of this.

    17. JG

      And it seems like with these things, the more context you get, the better the models work, right?

    18. KL

      Yeah.

    19. JG

      That's been the, the opening. I'm curious 'cause we were talking about model... switching between models and model portability and model cost, and like I gotta... I mean, I'm just curious, like I, I haven't seen a whole lot of difference between using like a, a world-class frontier model versus using the more commodity ones. I'm curious, have you guys seen that for, for, for doing a lot of this stuff with Second?

    20. KL

      Yeah, this is a good question. So if, if you say like, if you say, "I wanna analyze what is a semantically meaningful moment that happened on your computer." Let's say you go to a calendar and you move the invite. That you don't need, um, Fable.

    21. JG

      Yeah. [laughs]

    22. KL

      Um, uh, there are many tasks like that. Essentially, it's like a hierarchy of information you wanna understand. Um, a different kind of information would be like, "What is Joel's goal for this week?"

    23. JG

      Mm-hmm.

    24. KL

      That's something you want a very expensive model, but you don't want, you don't need to run it very frequently.

    25. JG

      Mm-hmm.

    26. KL

      So essentially, Second involves like a harness, it's like a harness design problem, like what, what models gets used when-

    27. JG

      Mm-hmm

    28. KL

      ... and so on. Um, and so that's where you have to build essentially this system to be like portable from the beginning. There are always gonna be more models come out. Like Jeff, for example, came out. It turns out that Jeff is s- extremely good at retrieval, right?

    29. MC

      Really?

    30. KL

      Yeah.

  11. 23:34 – 26:20

    How You Use Second (APIs and Apps)

    1. MC

      So, so maybe let's say es- uh, let's assume like someone's, um, you know, listening, like starts to use Second or kind of signs up. Like maybe talk a little bit about like the product, what they have access to, like, you know, something a little bit more concretely about like how, how, how a human interfaces with this-

    2. KL

      Mm

    3. MC

      ... what controls they have, et cetera.

    4. KL

      Yeah. So today the way it works is that you install an app.

    5. MC

      Okay.

    6. KL

      The app asks for some permissions.

    7. MC

      Right.

    8. KL

      Um, uh, and then you give the permissions, and then you specify what your tokens are, and then it just runs a bunch of... It starts running agents on top.

    9. MC

      Yeah.

    10. KL

      Uh, not-

    11. MC

      Yeah, I, I remember when I in- installed, I was like, you know, connect to Slack, connect to Signal, connect to WhatsApp.

    12. KL

      Yeah.

    13. MC

      You know, is that what you're talking about?

    14. KL

      Uh, not, not, not that. Uh-

    15. MC

      Oh.

    16. KL

      I mean, that's part of it.

    17. MC

      Yeah.

    18. KL

      But also the, the raw computer stream essentially.

    19. MC

      Oh, right. Yeah, yeah, yeah.

    20. KL

      Yeah. So, and, and the connectors help, you know. It's, it's like a sample efficiency problem. It makes it learn faster.

    21. MC

      I see. I see.

    22. KL

      Yeah. Um- Uh, s- and, and it starts running background agents, and there's pretty much nothing you gotta do from there-

    23. MC

      Yeah

    24. KL

      ... because the background agents keep, keep running, and then you start producing these, like, APIs out of, out of you, right? Like, there's a API about, uh, uh, uh, what, what you just did 10 minutes ago.

    25. MC

      Yeah.

    26. KL

      And there's a API for what are you gonna do, uh, for the next, next week, what are your priorities.

    27. MC

      Yeah.

    28. KL

      There's a API for your reminders, and there's a API for your wiki, like-

    29. MC

      Yeah

    30. KL

      ... people in your life, processes in your life.

  12. 26:20 – 30:43

    Toward Jarvis: AI Everywhere

    1. MC

      Are you planning to evolve the, the interface? I mean, you've just spent a lot of time talking about how it may evolve, but I always view Second as, like, out of view of an agent where you talk to it, you know, and, like, the UI is somebody else's problem. But are you also, like, actually anticipating innovating there?

    2. KL

      Um, I think it will be something that we kind of focus on in the, in the long run.

    3. MC

      Yeah.

    4. KL

      Um, because, like, uh, right now it's a systems problem and, uh, you know, the understanding problem comes first.

    5. MC

      Can I, can I just put in a request?

    6. KL

      Mm-hmm.

    7. MC

      Okay, so I want, I want to go to my house-

    8. KL

      Yeah

    9. MC

      ... and I want, like, that 1970s AI, [laughs] you know, like, the head on the screen that I can talk to [laughs] and, like...

    10. KL

      Yeah.

    11. MC

      You know, it's surprising. We have these agents, and they're so smart. Like, my Second is so smart. I, I just... Wow. And I'll ask it questions. It's very good, but it's not embodied, and it drives me nuts, man. [laughs]

    12. KL

      [laughs]

    13. MC

      So like... [laughs]

    14. KL

      Yeah.

    15. MC

      If in the future you can have Alfred actually be an Alfred, I'll be very happy.

    16. KL

      Yeah, yeah. I, I think we, we see a lot of this in, in the movies, right? Like, there's Jarvis. People talk to Jarvis all the time.

    17. MC

      But why, and why, why have not...

    18. KL

      We don't have Jarvis, and there's many aspects of Jarvis that people underlook. It's not just that it's a, it's intelligence and unders- like, it can do things.

    19. MC

      Yeah.

    20. KL

      But it's more that it's everywhere. Like, it's like-

    21. MC

      Right

    22. KL

      ... it's on your door. It's in your armor. It's, it's-

    23. MC

      Yeah

    24. KL

      ... you know, in, in your screen, and that kind of con- continuity and ubiquity, it's something, it's hard to-

    25. MC

      I, I, I'll say, I'll say, like, like, kind of my, my, my visions of the sci-fi future basically started when I started using Second. I'm like, finally there's something that, like, feels like it's got context. It feels like it knows me. Like, I can talk about it. It feels like it's a relevant conversation, and so I do feel like we're a bit ste- you know, a step closer to that. But, you know, I mean, we're, we're still tied to these devices, so-

    26. KL

      Yeah

    27. MC

      ... you know, I don't know if it's gonna be in a robot or it's gonna be, like, in a screen or, or whatever it is, but it would be actually nice to have this thing somewhat disconnected from my phone and my...

    28. KL

      Yeah.

    29. MC

      Yeah.

    30. KL

      Yeah. In the long future, I imagine that every person will have... Like, essentially everyone will has, have a Second, and all the devices just becomes inputs.

  13. 30:43 – 32:56

    Fort Knox Meets Delightful UX

    1. KL

      on-

    2. MC

      So I j- I just have one more question, which is, um- Uh, and, and, and, and Joel, Joel kind of touched on this. Like, you're like such a hardcore systems guy. Um, like, I mean, you know, you work... You know, who's your advisor, Percy?

    3. KL

      Percy Liang, yes, at Andreessen.

    4. MC

      He also worked with Jamon A., hardcore, you know-

    5. KL

      Yeah

    6. MC

      ... um, security guy. Um, and you know, a lot of the, the, the path that took you here was this kind of hardcore security stuff, and so you're building this pretty serious system. You know, it's very secure, um, and very functional. On the other hand, when I interface with it, it's like this lovely user experience, and so like, is there a bit of a schizophrenia in the company where you both need to be like this super hardcore, you know, build the Fort Knox infrastructure, but also make it very usable? Or does like the LLM basically solve, you know, the, the interface problem?

    7. KL

      Yeah, yeah. There, there are two kinds of problem we're dealing with, the infrastructure problem and the interaction problem.

    8. MC

      Yeah.

    9. KL

      And we've made our mistakes- [laughs]

    10. MC

      These are mostly, but they're totally different. [laughs]

    11. KL

      Like, very.

    12. MC

      Like you're as low on the stack as it gets.

    13. KL

      Yeah.

    14. MC

      I mean, that's not hardware.

    15. KL

      Yeah. So we, we, we made a lot of mistakes on the interaction side. We tried to go all in on interaction, and turns out it's actually very hard. There's a lot of details that go in-

    16. MC

      Yeah

    17. KL

      ... you know, to create that delightful experience.

    18. MC

      Oh, you mean like actually creating a UI?

    19. KL

      Yeah.

    20. MC

      Yeah, that's right.

    21. KL

      UI, but also like, you know... Yeah, yeah, like what kind of like personal pages you can create on top of this. I mean, and we realized that, you know, like it's, it's something that would take a long time to get right, and this is a time like everyone would take this kind of like understanding of themselves and build this. And so we want to enable people to do that instead of doing it ourselves. And, um, so we decided to like close down the surfaces that we have. There's only texting, there's only API, and there's nothing else.

    22. MC

      Right.

    23. KL

      You can technically use it as a full agent, like a co-work agent on, on your, on your laptop, like it runs Bash and all that.

    24. MC

      Yeah.

    25. KL

      Um, but we meant it as like very minimal-

    26. MC

      Got it

    27. KL

      ... so we can like make the interaction like more, more smooth.

    28. MC

      Ken, thanks for coming in. Um, congratulations on everything, uh, and the launch. And is there anything that you want to, uh, leave the listeners with?

    29. KL

      Go try it out today. Um, it's available, uh, it's on 2.computer. Uh, it's a very short URL, and we hope to, like give you access as soon as possible. And if you resonate with this, um, mission, you know, come join us. There's a lot to build.

    30. MC

      Great. Thank you so much.

Episode duration: 33:10

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode kBJe4H_WobA

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.