Complete Course: AI Agent Products (with Warp.dev CEO Zach Lloyd)

1. 0:00 – 2:00

   Intro

   1. AGAakash Gupta0:00

      Warp is adding 1 million ARR every 10 days

   2. ZLZach Lloyd0:04

      The rough size of our user base, which is, I think as of this month, there'll be close to 700,000 active developers who are using Warp.

   3. AGAakash Gupta0:12

      AI agents, and specifically AI coding agents, are the number one trend in product development right now.

   4. ZLZach Lloyd0:17

      One very, very powerful new primitive, which is intelligence.

   5. AGAakash Gupta0:21

      How do you build a good agentic product? How do you code well with AI agents? Some people are making tons of money. Most people are getting lost in the shuffle. I'm so excited to share this conversation with CEO and founder of Warp, Zach Lloyd.

   6. ZLZach Lloyd0:33

      What's particularly challenging, and you'll see this with Warp and with Claude and Cursor, and, like, everyone is trying to figure out how to price this stuff. And what's cool about this is when I go into this, I immediately get into the agent flow. Now, I'm asking, I'm, like, s- probably for the first time, I'm seeing, like, "Oh, wait a second, Warp can just, like, use an agent to fix my thing for me?"

   7. AGAakash Gupta0:52

      So, as I previewed at the beginning, some people are making money with agents. You guys have figured out how to do it. Others are lost. So, who's actually winning, and what business model innovations work for agentic products?

   8. ZLZach Lloyd1:04

      The typical SaaS pricing mechanism of a fixed price per seat subscription, I think it doesn't work that well.

   9. AGAakash Gupta1:12

      Wow, more than 19X growth in a single year, so you're really in that explosion that's happening.

   10. ZLZach Lloyd1:17

       If you are in a business where you are less about, like, improving productivity, but you have a more measurable outcome where you charge for much closer to the value that's being provided.

   11. AGAakash Gupta1:29

       How big is Warp? What revenue and usage numbers can you share with us? Really quickly, I think a crazy stat is that more than 50% of you listening are not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify Podcasts, my commitment to you is that we'll continue to make this content better and better. And now on to today's episode. Zach, welcome to the podcast.

   12. ZLZach Lloyd1:57

       Thanks for having me, Aakash. It's awesome to be here.

2. 2:00 – 8:27

   How big is Warp

   1. AGAakash Gupta2:01

      How big is Warp? What revenue and usage numbers can you share with us?

   2. ZLZach Lloyd2:05

      [lip smacks] So, I can share our growth rate. So, as you said in the intro, we're adding, it's now over a million dollars, uh, in ARR every 10 days, and that's accelerating. And that's up, you know, we, we just started monetizing Warp, um, like, basically we're, you know, close to nothing at the start of the year, and so it's been very, very cool to see the acceleration. And then the other thing I can share is just, like, the rough size of our, um, of our user base, which is, I think as of this month, there'll be cl- you know, close to 700,000, uh, active developers who are using Warp, which is pretty cool, and that's also growing quite quickly.

   3. AGAakash Gupta2:44

      Insane numbers. And revenue is up 19X this year?

   4. ZLZach Lloyd2:48

      Yeah. Or more, I think, at this point. Yeah.

   5. AGAakash Gupta2:51

      Wow, more than 19X growth in a single year, so you're really in that explosion that's happening. We're seeing it with some other tools similar in the space, whether it's Lovable or Replit. What was the turning point for you guys from almost nothing to now this explosion?

   6. ZLZach Lloyd3:08

      Yeah, so it, it really had to do with, um, finding the right interface, uh, for AI within the product. And so the, you know, the history of Warp, just very briefly, is we started off as a, you know, re-imagination of the terminal, so of the command line UI. And we spent a lot of time, you know, building what we thought was, like, the most usable and accessible and powerful version of just doing command line work. And it turns out that that, um, that interface is incredible for doing agentic work. And you started to see that at the beginning of this year with all of these, like, um, CLI-based agents that have come out, so things like Claude Code or Gemini CLI. And, um, you know, we had actually, even before those tools came out, started to make it so that instead of using Warp as a terminal, you could use it to run agents. But it wasn't until we, like, really went fully in on the agentic features and made them front and center, and even repositioned Warp from being a terminal into what we now call an agentic development environment. Uh, and we did that a lo- uh, uh, that launch in June, where we really started seeing the growth s- like, accelerate super quickly.

   7. AGAakash Gupta4:19

      Wow, that's the power of building AI agent products well. You've been a CTO multiple times. You worked at Google-

   8. ZLZach Lloyd4:27

      Mm-hmm

   9. AGAakash Gupta4:27

      ... for years, so you are very familiar with PMs and founders. For any PM or founder watching in 2025, how critical is understanding agentic AI? Are we at a learn this or get left behind moment?

   10. ZLZach Lloyd4:42

       I think so. I mean, [chuckles] I think, uh, it would be hard for me to imagine building a product today and not, um, thinking about how you can apply this technology to, to whatever problem you're solving. Uh, I would just be, like, surprised if there's a problem out there that is a software problem where it wouldn't benefit from some amount of intelligence. And that's, like, really what, what I think of as, like, the thing that's changed, is that there's this sort of new primitive available, where if you're someone building a product, um, you now, you know, it used to be, like, if you're building a product, you could sort of pick, like, uh, a database or external APIs that you want to rely on or just, like, a whole technology stack for building it. And now there's this one very, very powerful new primitive, which is intelligence, that you can have in your app. And so whether you're building a productivity app, you're building a consumer app, uh, you're building something that's much more on, like, you know, for helping go-to-market people, like, I think every kind of app out there at this point would benefit from having, um, agentic features.

   11. AGAakash Gupta5:50

       So, let's get tactical, right?

   12. ZLZach Lloyd5:52

       Yeah.

   13. AGAakash Gupta5:52

       I promised our audience we'd reveal how to build good agentic products.

   14. ZLZach Lloyd5:55

       Yep.

   15. AGAakash Gupta5:56

       Most people are building AI products that feel like gimmicks. [laughs]

   16. ZLZach Lloyd5:59

       Right. [laughs]

   17. AGAakash Gupta6:00

       So, what's your framework for identifying where AI agents add value versus where they're just tech for tech's sake?

   18. ZLZach Lloyd6:07

       Yeah, so-I think it's, like, the same fundamentals of product development as if you're building an app in the world before agents apply if you're building with agents. And so y- you need to start with the problem, in my opinion. So, like, what problem are you trying to solve for a user, for a customer? Um, you know, is it a, is it a deep problem? Is it a nice-to-have problem? Um, once you've kind of identified that there is a problem that you wanna solve, uh, and in Warp's case, that problem is, like, it takes a ton of time and is very expensive and hard to develop software. It's, like, a very, like, kind of general problem. But, like, the, the problems like that exist in, in a ton of different domains. Uh, you start with the problem, and then I think you start to explore the, the s- the solution space, and you need some hypothesis around, like, okay, um, you know, I think, I think that if, uh, y- you know, I'm trying to, to solve this problem for some user, that if we inserted intelligence in this particular workflow, I might make that workflow faster, and that would be useful for people. Um, and so, like, I'm trying to think of a good hypothetical without just totally focusing on Warp. But, like, let's say I'm building a calendaring app or something. It's like, it's probably useful to have some intelligence with, uh, when meetings are scheduled. It's probably useful to have an LLM be able to, like, understand, um, you know, the patterns where you like to work or your coworkers are available, uh, and s- and, you know, factor that into the scheduling algorithm. And it used to be that for something like that, you would have to, like, code it as an algorithm. And what I mean by that is, like, you would probably set up a bunch of, you know, specific rules. Like, this person likes things in the morning, that person likes meetings in the afternoon, this person doesn't want a meeting with more than five people or whatever. And what's different these days is, like, you don't necessarily have to write an algorithm. You can simply provide a bunch of context, um, to an agent, and the agent will give you an intelligent answer. And so, you know, as you survey, like, the space of solutions for whatever problem you're solving, you have this incredibly powerful new way of solving the problems. But you still have to start with, like, a problem that actually

3. 8:27 – 14:53

   How he made Warp

   1. ZLZach Lloyd8:27

      matters.

   2. AGAakash Gupta8:29

      Walk us through your own development process.

   3. ZLZach Lloyd8:32

      Yeah.

   4. AGAakash Gupta8:32

      We know that this was the inflection point, right? How did you decide how to, what to make agentic and how to package it in this way?

   5. ZLZach Lloyd8:41

      Yeah. So we actually went through a number of iterations with AI and Warp. Uh, I think this is, this is instructive 'cause I think it follows kinda like the market as a whole. So even before, um, LLMs and ChatGPT came out, we actually had some AI features in Warp. We had one that would let you, like, sort of type in English, and in real time it would translate that English into a command. And, like, that's kinda the most obvious, like, way of doing it. Like, so, like, it's, you know, if you're, if you're a terminal user, you'll realize it's hard to remember commands. Like, how do I search for files on my computer using a command? And so we're like, "Okay, well, here's an easy solution to that. You type the command you want." You don't type the task. You type the command, and you're like, "I wa- what's the command for finding files?" And it gives you back this thing that's like find-name.- like whatever. Um, and so we started with that, and then when ChatGPT came out, we were like, "You know what? It's, there's, like, a more general purpose thing that would be useful here, uh, where you can just, like, ask an agent questions and get answers." And so, and we're like, "You know, we'll make it really easy to provide context from your terminal session," uh, and so we put a chat panel into Warp. And I, I think this is, like, where a lot of apps started. It's like, "Well, okay, it's ChatGPT. It's obvious you can have a conversation with it. Uh, you know, what's, like, a cool, lightweight integration we, we can, you know, we can get this into our app?" And so we did a chat panel. And then w- the more we thought about it and looked at it, we were like, "This is not a native, native integration to the app." Um, and for our case, there's actually, like, you know, a much sort of, uh, more native way of integrating the AI, and this is what I would encourage people to think through, which is the terminal or the command line is already, uh, oriented around, like, it's an interface where you tell the computer what you want it to do, and you just tell the computer what you want it to, to do in, um, in terminal language, in computer language. But with LLMs, it turns out you can actually just tell it what you want it to do in English. And so that was, like, the big unlock for us from a product standpoint is, like, um, you know, take the native interface of the command line, which is a very powerful interface for, like, executing things, and just have it execute your English essentially. And that, uh, mindset shift, which, you know, we did, like, a little over a year ago. We actually did it way before all the, the CLI coding agents, and we actually called it agent mode, which has become a very-- We, we were the first to do that, which has become, like, a kinda standard name for these features.

   6. AGAakash Gupta11:19

      Yeah.

   7. ZLZach Lloyd11:19

      Um, that was, like, when we first started our, getting our, our initial revenue traction 'cause people were like, "Okay, this is cool. I'll tell my computer to do things." Um, and that insight, that, like, very native way of interacting with AI has been just, like, this front door to all of this other stuff we can do. Because once you have this, um, you know, way where you're telling the computer what to do, it becomes a question of, like, how do you add then the right tools so the computer can actually do what you're asking it to do? And so for us, at first there was only one kinda tool. So in Warp, if you asked an agent to do something, it could just run terminal commands, and that's actually pretty powerful because terminal commands are very flexible. But since then, we've started to add a whole bunch of other tools, so it can, like, edit files for you. It can, like, read web pages. It can read files from your file system. And so w- but, but it all took that, like, main thing of like, okay, what's the rightentry point in the app to actually interfacing with an agent in a way that feels natural, and that's been our, our biggest product unlock.

   8. AGAakash Gupta12:27

      I think talking to a lot of people who are building AI agent products-

   9. ZLZach Lloyd12:31

      Yep

   10. AGAakash Gupta12:31

       ... this is always the thing. There's always this UX challenge around how to deploy the agent correctly for-

   11. ZLZach Lloyd12:38

       Yeah

   12. AGAakash Gupta12:38

       ... your workflow.

   13. ZLZach Lloyd12:39

       Yes.

   14. AGAakash Gupta12:39

       So how would you extract out the learnings you had for, like, a general purpose? Like, how do I solve these UX challenges around agentic products? What is the right process for me to really deliver an agentic experience that feels integrated?

   15. ZLZach Lloyd12:54

       Yeah, um, great question. I can just think of some other examples that I think are good to maybe, like, inform this a bit. Um, like, there's a, a new set of spreadsheet products that I think are interesting. And so I, I used to work on, uh, I used to lead the engineering for, for Google Sheets when I was at Google, and so I've always been, you know, for a long time now, very interested in spreadsheets. And the, um, the, like, f- again, the first iteration of AI that I saw in, like, Google Sheets was, like, this Gemini button that, like, sits off in the corner, that pops open a chat panel, and it's like, it's almost like, "Don't do that." [chuckles] And, and what I, what I have seen recently, which I think makes much more sense, uh, is the ability to have AI populate cells. And so you're, like, taking the, like, fundamental UX of, like, what the, what the app was already for, which was, like, um, you know, working with data, working with lists, and you're having the insight that, okay, one of the, like, like, you know, challenging things about working in this interface is either getting data into it, so you can use AI for that, meaning, like, I wanna have a whole column of, uh, where the AI is bringing data into it, or it's working with the data that's already in it, and, like, again, you can work with that data by learning a cr- a ton of crazy spreadsheet formulas, or what the AI enables is, like, you to go one level up in abstraction and just say what you want in English. And so any place where the sort of, like, prior solution involved, um, expressing something complicated, uh, using, like, some sort of, you know, either spreadsheet formulas or code or, like, a scheduling algorithm, I think that's where you should be looking for, like, okay, is there a native way that I can do this with an agent?

4. 14:53 – 15:42

   Ads

   1. AGAakash Gupta14:53

      Today's episode is brought to you by Vanta. As a founder, you're moving fast toward product market fit, your next round, or your first big enterprise deal. But with AI accelerating how quickly startups build and ship, security expectations are higher earlier than ever. Getting security and compliance right can unlock growth or stall it if you wait too long. With deep integrations and automated workflows built for fast-moving teams, Vanta gets you audit ready fast and keeps you secure with continuous monitoring as your models, infra, and customers evolve. Fast-growing startups like LangChain, Writer, and Cursor trusted Vanta to build a scalable foundation from the start. So go to vanta.com/aakash. That's V-A-N-T-A.com/A-A-K-A-S-H to save $1,000 and join over 10,000 ambitious

5. 15:42 – 16:37

   Why Most AI Agents Fail at Launch

   1. AGAakash Gupta15:42

      companies already scaling with Vanta. Today's episode is brought to you by the experimentation platform Kameleoon. Nine out of 10 companies that see themselves as industry leaders and expect to grow this year say experimentation is critical to their business, but most companies still fail at it. Why? Because most experiments require too much developer involvement. Kameleoon handles experimentation differently. It enables product and growth teams to create and test prototypes in minutes with prompt-based experimentation. You describe what you want, Kameleoon builds a variation of your webpage, lets you target a cohort of users, choose KPIs, and runs the experiment for you. Prompt-based experimentation makes what used to take days of developer time turn into minutes. Try prompt-based experimentation on your own web apps. Visit kameleoon.com/prompt to join the wait list. That's K-A-M-E-L-E-O-O-N.com/prompt.

6. 16:37 – 19:35

   UX process on an AI agent

   1. ZLZach Lloyd16:37

      Uh, and what I, what I would think is, like, not necessarily the right path is, um, the, like, let me put chat in my app [chuckles] path. 'Cause, like, uh, you know, I think the chat, chat is like, like ChatGPT has that, and the browsers have that, and, you know, Siri is gonna have that. And, like, an overlay of chat is, like, a very kind of thin differentiator or moat. But something that's very deeply, like, native to the UI of your app, um, which uses intelligence and might not look exactly like chat, but lets a user express their intent in, in English or in images or in video can... is a really powerful way of, like, getting the agent natively to help you do whatever your user's trying to do.

   2. AGAakash Gupta17:26

      And I think this speaks to why AI agents are so powerful, actually, is because they're allowing us to realize the full vision of AI.

   3. ZLZach Lloyd17:32

      Yeah.

   4. AGAakash Gupta17:32

      Where we, we always felt like chat was a limitation on the UX. The agent, you're giving the LLMs tools. You're giving it the ability to plan, think, react.

   5. ZLZach Lloyd17:42

      Yes.

   6. AGAakash Gupta17:42

      That is allowing you to embed it into your actual normal workflows.

   7. ZLZach Lloyd17:47

      Yeah. I think that's right too. So, like, another way you can think of it is, like, w- what would you have a human doing for this job? Which is, again, it's a crazy new thought as a, as, like, someone who's developing products. And you could be like, "Okay, well, if, uh, if you had a human to assist you or to assist your user in doing the task, what would be the most useful way that the human could do it?" And it could be that the human just, like, sits over your shoulder like a co-pilot and tells you what to do, but I think, uh, a lot of times you'd wanna put the human to, like, work. And, um, you know, agents are kind of those virtual... workers where if you can find a way that they can, like, do the job in your app, that's gonna be really powerful. Um, and you can see this, like, uh, I think if you look at the sectors that have the best product market fit right now for agentic products, it's, like, coding products, like Warp, and it's, uh, where the agent is helping write the code, which is what a person might do. And it's, like, customer service products, so companies like Sierra or Fin or I think Decagon. It's like, and you're, you're literally for the first time ever, it's like you can have an agent do something that only a, a human could do before, which, you know, has, like, some... You know, I, I share concerns about, like, replacing people's jobs. That's not where, where, uh, you know, we're not trying to do that. But, like, as a business owner, I think, um, just having the ability to, like, be like, "Okay, I could have a something of human-like intelligence use my tools and help my users," that's really valuable.

   8. AGAakash Gupta19:30

      So we're talking about the UX of agents, how to integrate it into your product.

   9. ZLZach Lloyd19:34

      Yeah.

7. 19:35 – 29:32

   Live Demo: Building AI Agent with Warp

   1. AGAakash Gupta19:35

      There's really no better way to learn than to see, feel, do.

   2. ZLZach Lloyd19:38

      Yeah.

   3. AGAakash Gupta19:38

      So can you show us in action, how should people be using the Warp agent?

   4. ZLZach Lloyd19:43

      Yeah. So when you open up Warp, uh, you'll see something that looks, um, a lot like a, a command line. And again, if you're a developer and are familiar with the command line, you'll find it pretty, uh, you know, pretty, pretty understandable, that you can just kinda come in here and run terminal commands. Um, but the big difference in Warp, and the thing that's been the, the biggest product unlock for us, is that in addition to typing terminal commands, you can just, um, directly, uh, type agent queries into, uh, into the same exact input. So I could be, it could be as simple as something as, like, "Tell me about our current repo." I'll just kinda show you how this works. So this is detected as being, uh, English rather than a terminal command. When you type something like this, it starts a process of exploration where the agent starts using the tools that it has access to to try to, in this case, answer my question. Uh, you can see that the primary tools that, uh, Warp has access to are basically terminal commands, but it also has access to a bunch of other stuff that I'll show you too. So here, this is like, how would I learn about a new project? I would just ask about it. Um, before a tool like Warp, I don't, I don't n- really know what I would do. I'd be looking for, like, a README. Um, but this lets me-

   5. AGAakash Gupta21:07

      Yeah

   6. ZLZach Lloyd21:07

      ... sort of interactively, like, explore the code base as I want. Now, I think it's cooler if I can actually show you how I would implement a feature or make a coding change using Warp. And again, this is accessible f- not just for developers. Um, I know there's a, like, a significant, like, PM audience that, uh, listens to this podcast. Um, you, you don't have to even really know how to code to do it. It'll be helpful if you do, but you don't have to. And so, the, the feature that I'm gonna build is actually a feature in Warp itself, uh, which is something that probably should already exist but doesn't currently, which is, you'll see when I hover over these buttons here, I get tooltips. That's nice. When I hover over this, uh, our directory picker, I don't.

   7. AGAakash Gupta21:51

      Mm.

   8. ZLZach Lloyd21:51

      And I feel like that should also have a tooltip, 'cause it's like you, you don't even know that you can click on it. You can click on it and navigate around and change your directory. So what I'm gonna do is have Warp implement that for me. And so, the way that I'll do that is I'm actually just gonna speak to Warp, and this is how I use agents quite often now. And I'll just reference, uh, some context. So for instance, I'll say, "Can you add a tooltip to our directory picker context chip that says, 'Choose a directory.' Take a look at the attached screenshot for, uh, more context, and I'll tell you what file, uh, I think it should be in as well." So I speak to Warp, right? I just say that. I'm gonna go ahead and just take a screenshot of this. So what I'm doing here is I'm adding context for the agent, and then I'll also give it a file. Uh, so I can just hit the At key here and reference a file. Um, I don't really like "Choose a directory." I'm gonna change this to something like, um, "Switch directory," um, and-

   9. AGAakash Gupta23:03

      I love using voice dictation for this too. [chuckles]

   10. ZLZach Lloyd23:06

       I, this is literally what I do, as long as I'm not in a public place, uh, is that I, I just talk to my computer, which is, like, it's so crazy how quickly we've gone into, like, sci-fi and people, I'm not sure everyone appreciates it. Um, so I'll do this, and as I do that, it starts figuring out what I wanna do. And I've just given it, like, the, uh, the sort of output that I want here. Um, these models are super smart and can learn your code base. And in our case, this is, like, a very nontrivial code base. Um, it's over a million lines of code, and it's in a, you know, kinda hard language that we use here called Rust. And so, it's figuring it out. It's reading all of this. It uses these tools, and then it's gonna show me, um, the code change here. So it's creating the code change right now. We'll give it a second. Oh, still thinking about it. Give it one more second.

   11. AGAakash Gupta24:08

       I love watching these things think. It brings me back to my coding days. [chuckles]

   12. ZLZach Lloyd24:13

       Yep. And then, you know, this is, this is weirdly a little bit of, like, um, you know, what, um, coding is like when you're coding with an agent like Warp, is you could just end up sitting here watching it. And so, what I will typically do because of that is I'll do multiple things at once, which I could also show you. But here we go. So we have, um-Basically, it's showing me the code change. Um, I know this code. I think this is basically right. I'm gonna... I could go in here. If I wanted to, I could actually go ahead and edit this, but I'll just accept it. And then in Warp, we have some really nice features that let you track what the agent is doing as it's doing it. So, I can just pop this open over here. And if I wanted the agent to actually go in and change it, I could sort of like, you know, almost do, like, a code review on the agent's, uh, work. Uh, but now it's made the change. It's building it for me over here. Again, I haven't touched it. It just knows that after it makes a code change like this for me, that I generally want it built. Um, so this'll take a second. And-

   13. AGAakash Gupta25:18

       It's crazy that it knows that, though, actually, 'cause we didn't instruct it.

   14. ZLZach Lloyd25:21

       We didn't instruct it. But, again, if you're, if you're working with these agent tools, one of the product challenges that you're gonna wanna try and solve is, like, how do you make it so that as a user, the user doesn't have to keep repeating themselves? Um, and so there's different techniques for doing this that different products use. So Warp uses, you know, a combination of rules. So rules are sort of, like, persistent context, um, that the agent, uh, has access to. So I, I may have a rule around, uh, after you make a code change, build my code. I c- can't remember if I have that or if it's just smart enough to do that. Um, another one is, like, you know, you have concepts around, like, saving and reusing prompts, which are really useful. Um, I'm gonna actually see if I, I d- I do. I'm gonna actually just run this directly and just go right to running rather than waiting for it to build. Uh, this is gonna take a minute. It's possible we'll wanna f- sort of not want people to sit here watching my code build, but if you... [laughs] This is what it is like to develop these days. So this is building. Um, yeah. And so I think figuring out this sorta, like, memory or learning or h- how do you make it have the right context is a, is a really cool product challenge. And it's also one of the things that you can do as a product person working in this space to make your product stickier, because the more that it learns and understands your users, uh, you know, the, the, the more sort of like, um, uh, the more they'll wanna stick with it, the less they'll wanna use some tool that doesn't know them as well. So this is building. Give it a second.

   15. AGAakash Gupta27:03

       And what does that mean, like, sort of, uh, tactically and operationally, like, making sure that your agent is, like, understanding your users? Like, what are the things that you're building into your product to make sure that it's doing this correctly?

   16. ZLZach Lloyd27:16

       Yeah. So there's, uh... So one thing, which I'll, I can show you while this is building here. I'll go ahead and close this. Um, we have this thing called Rules, for instance. And so Rules, can you see over here? Um, so here's one. Ask me before committing changes. So, uh, you know, agents kinda have non-deterministic behavior where they might start committing changes. They could commit them to the wrong branch. I just want our agent to be very explicit. Um, I want it to always format my code before it pushes it to origin. And so this is, uh, a way of, like, giving, like, persistent context to the agent so that it knows me. And we have something, I think, which is cooler than this even, which is, like, one step up, where we will suggest these rules for people. Um, so let's see. So here we go. It built it. Do you see it?

   17. AGAakash Gupta28:09

       Yeah. Wow.

   18. ZLZach Lloyd28:11

       [laughs]

   19. AGAakash Gupta28:11

       We have the tool tip, like, literally.

   20. ZLZach Lloyd28:12

       We have the tool tip. And I could be like, "You know what? I want it left aligned." Like, I could, I could... A typical flow for me, and this is how I do all of my development these days, is I just, um, start with a prompt, and then I look at it and I'm like, "You know, this looks pretty good, but maybe it could be better." Uh, and then I will re-prompt and reiterate until I have the agent get me to a point where I'm happy with it, and then I will have the agent actually review its own code, and then I'll have the agent, uh, you know, uh, basically push the commit so someone else on my team can look at it. But I'm, like, very, very tightly, um, coupling, you know, with it to get to a, to get to a point where it did what I want. So anyhow, this is quick, uh, quick kinda overview of Warp. Anything else I could show you in it?

   21. AGAakash Gupta28:59

       That's amazing. So how does Warp use Warp? What are the advanced tactics people can use to supercharge their productivity?

   22. ZLZach Lloyd29:08

       Yeah. Um, do you want me to, to stop sharing or just kinda keep this-

   23. AGAakash Gupta29:11

       Keep sharing. Yeah.

   24. ZLZach Lloyd29:12

       Keep sharing? Okay.

   25. AGAakash Gupta29:13

       I think for, like, this whole, like, next four or five sections, even if not much is going on on screen, we'll just keep the screen share.

   26. ZLZach Lloyd29:19

       Okay. I mean, if you want, I can also, like, have other things going in the background. Um-

   27. AGAakash Gupta29:22

       Yeah. Well, I think, like, one of the things you can talk about is, right now, is, like, having multiple agents doing things at once, so that-

   28. ZLZach Lloyd29:28

       Yeah

   29. AGAakash Gupta29:28

       ... that's something you can demo, for instance. Today's episode is brought

8. 29:32 – 31:05

   Ads

   1. AGAakash Gupta29:32

      to you by Amplitude. Building great digital products is hard. You know that better than anyone. Getting teams aligned, measuring what matters, and scaling your product strategy isn't easy. But what if you had a clear framework to guide your next steps? That's exactly what Amplitude built. They studied the best product teams to understand what really drives impact, and turned those insights into the Digital Experience Maturity Assessment. In two minutes, you'll be able to see where your team stands and what you can improve to build better products faster. Click the link in the caption to take the free assessment and get a clear path to product growth. AI Evals are one of the most important skills for PMs, and I know you know they matter. The question is, are you doing them right? Most teams are winging it with basic metrics and hoping for the best. Meanwhile, the teams that actually ship reliable AI, they've cracked the code on systematic evaluation. Today's episode is brought to you by the AI Evals for Engineers and PMs course by Hamel Hussein and Shreya Shankar. This live Maven course will teach you the battle-tested frameworks from Hamel and Shreya, who are the engineers behind GitHub Copilot's evaluation system and 25-plus production AI implementations. Four weeksLive instruction. Next cohort starts July 21st. Start shipping AI that actually works. Enroll at maven.com with my code ag-product-growth for over $800 off. That's ag-product-growth.

9. 31:05 – 38:25

   Systems That Drive Adoption

   1. ZLZach Lloyd31:05

      Yeah. So actually, so how about before I answer this question on how Warp uses Warp, let's say I wanna, um, actually be doing, you know, other, uh, you know, multiple things with agents. So I have this, this agent here that just did this tool tip. Um, let's say that I also want to, um, have an agent kind of go and like fetch, uh, yesterday's feedback from GitHub. So this is like a kind of cool thing 'cause it's not about necessarily building an app, but it's like, um, just showing how general purpose agents can really help your workflow. And we have like a lot of people who use Warp for this kind of thing. So I would be like, "Can you use the GitHub CLI to fetch yesterday's new GitHub issues and sort and summarize them, pull out themes for me?" Y- this should be, uh, the Warp public GitHub repo. So I'll start this agent and then I'll answer your question. So, uh, Warp, uh, at Warp, we are using Warp to build Warp, which I, again, I think if you are building, uh, an AI, um, an AI product, you really wanna dog food it. And so dog food's like a core principle of, uh, you know, how we get the product to be something that's useful. Um, we, we and me specifically, I've set up a bunch of guidelines, which are, um, it's sort of like, we call it the coding mandate, where we ask for, uh, Warp engineers to start every coding task with a prompt. Um, and they don't necessarily have to finish it with a prompt because w- the sort of state-of-the-art where we're at with agentic development right now is like there are definitely a class of problems which are too complicated or there's-- you could just do it faster if you just do it on your own. Um, and, uh, so we don't-- I don't force like getting all the way to a completed task with a prompt. But what we do ask then is like for folks, after they start with a prompt, if they're not able to finish the task, to share feedback. And so that feedback just goes into one of, we have like five Slack channels, which are for different kinds of feedback that developers encounter as they're trying to build Warp using Warp. Um, and it could be like, "Hey, you know, the agent got confused doing X," or, um, you know, the, the agent, uh, you know, I tried to sort of like do a code review with the agent and it didn't do it. And so people are constantly sharing this feedback, which I think is really important. And conversely, we ask people to also share when it is able to do it, which is an increasing percentage of the time. We have this channel called, um, it's called Warped It, where, uh, we ask people to share looms of like, you know, show, show the agent doing a completed task. And then what's cool about that is that becomes content which can be shared out to the world, whether it's on our social, or sometimes we'll put these on our YouTube channel. And it's really, really powerful as a way of educating current users, prospective users, just people who are interested in general in terms of agentic development on like how to do it successfully and what the, you know, what the key aspects of like the prompting and the context are to get to something that works. And there is, by the way, there's like, there's quite a bit of skill in it. Like, you know, you need to, to, you know... Actually, like the, the best thing is like if you could tell the agent how to do the task, not just the outcome that you want. Um, but like there's skill in what context you provide. There's skill in like, can you come up with a good plan with the agent? And it's very similar to like working with a kind of junior engineer on the team. And so, um, showing these examples, and people are really interested in the world right now in terms of like, how do I, how do I use agents to do my own job? So there's a lot of, uh, appetite for that type of content.

   2. AGAakash Gupta35:18

      Love that. So how does Warp compare to the competition like Cursor, Claude Code, Gemini CLI?

   3. ZLZach Lloyd35:26

      Yeah. So that, you, you, you said the competition correctly. Um, so there's sort of two buckets of competition that we look at. So there's, you know, apps like Cursor or maybe Windsurf. Um, I don't know what Windsurf's current state is. But like those are IDEs that, and they tend to be forks of, uh, VS Code. And so they actually all have a very, very similar interface where you, you know, most of the real estate of the app is a, something that looks like Microsoft Word. It's like really geared towards, um, you know, writing code by hand. And then you'll have a chat panel on the side, and that chat p- that chat panel can do stuff obviously. It's not just like a read-only chat panel. But it's still, um, you know, to my point earlier, it feels to me, and I'm biased here, but it feels like a kind of a bolt-on experience to an old style app. Uh, and then the other, um, type of app that is really catching on right now is a pure CLI app. And just to show like, I think for, for people watching this, it could be interesting to see. So the pure CLI app is, uh, something like Claude, where you, you run it. You can actually run it from within Warp, and it has like a similar interface to Warp actually, where you get like this input box and you can tell it to do stuff. But because it's a terminal app and not like the actual terminal, you don't get, um-The sort of same, the same type of user experience that we can provide, where you actually have a GUI that, like, will, you know, show you all of your changes, and, like, lets you, like, search for files to open and edit files. And so, um, it's a, it's like Warp in, in a similar interaction pattern, but it's not as rich of an experience. And so, you know, the way that we are trying to differentiate is by, um, building a totally different kind of app. Like, like I, you know, sort of said earlier, it's like we call it an agentic development environment, where we are basically trying to, to support what we think the new workflow should be from ground up product principles. And so, I think the new workflow is, like, as a developer, you start with a prompt, but you probably wanna do some amount of hand editing for when that prompt gets stuck. You wanna be able to quickly see all of the diffs. You wanna be able to, like, reference those diffs, uh, easily for a feedback loop. And so, Warp is kind of in its own class here in terms of where it sits to these CLI apps and IDEs, which is very, very cool and I think is a big opportunity. But also, like, it isn't-- There are product challenges and trade-offs, just to be, just to be clear about that. Um, but yeah, that's how Warp is kind of neither a CLI app nor a, an IDE. It's like its own, its own thing. Does that make sense?

   4. AGAakash Gupta38:23

      Yeah. And I think there are a lot of really

10. 38:25 – 46:15

    Workflow to build AI Agents

    1. AGAakash Gupta38:25

       good nuggets in there for people trying to build agentic products, right? You don't wanna build a me-too product that all of a sudden the foundational model company is just creating and you build up for.

    2. ZLZach Lloyd38:33

       Exactly. [chuckles]

    3. AGAakash Gupta38:35

       You don't wanna create something that, uh, is just a bolt-on of an old interface that just feels like you shoved a chatbot in. Like, the whole lesson we're trying to teach people is you wanna think from ground up principles. What is the new workflow with agents? And then build for that new workflow.

    4. ZLZach Lloyd38:53

       I mean, that, that is exactly how I, uh, how I express it, and how, how I think about it, and why I'm, like, bullish, um, on our position in the market here. And I, I think, you know, if you're going into a very, very competitive, crowded space, like AI coding, I think differentiation is a, is a m- very important thing that you should be thinking about. Um, and so I, like, I do think Warp is, is one of one in our approach here. Uh, we are... Yeah, we're not the other things. Now, there's, again, there's challenges with that as well, but I think if I'm working on something that I wanna be working on for a really long time, to me, it's like the coolest and most powerful and most useful thing that we could do is actually try to build the best possible app for supporting what this new workflow is.

    5. AGAakash Gupta39:45

       So how did you think about onboarding people into this workflow? Like, what are the, what are the areas where you architected that to make sure that people actually activate? Because that's almost always the most important experience, is that first run product experience.

    6. ZLZach Lloyd39:59

       It's an awesome question, and has been, um, something that we've iterated on a ton throughout Warp's history. Um, I can tell you one thing that was, like, a big unlock, and actually I can show it, because I think it's cooler to show it. Um, [lips smack] so the principle... Let me clear my screen here. The principle, uh, was, so we, we had a lot of people who were downloading Warp, intending to use it as a terminal, which is fine. Like, that's what Warp was, like, early in its existence. And we were like, "How do we get people to experience an aha moment with Warp as an agent?" And so, I'll, I'll tell you things that I, that didn't really work that we tried, and then I can tell you things that have worked that we've tried. Um, [lips smack] so stuff that just doesn't move the needle in my experience is, like, like, a kinda warm welcome screen or, like, a guided product tour. Um, maybe it works for some amount of user education. Um, [lips smack] uh, you know, those tours with, like, the flashing lights that take you, like, here and here and, like, all across the product. Though that didn't really work for us, nor did, like, changing any copy really work for us, or putting different buttons or entry points on the screen didn't really work for us. What worked for us, uh, is something that, um, catches a user at the moment where we think our agent could be useful and helps them accomplish a task with the agent. So let me see if I could give, like, find a way to give a good demo of this. So if I were to, like, go over here, and let's see if I do this. Yeah. Perfect. So I tried to check out our master branch here, and I got an error, right? And it says, like, "If I did this, it would be overwritten by checkout." And then it gives me this message, "Please commit your changes or stash them before you switch branches." And so, like, if I know what I'm doing as a developer, it's like, "Oh, okay, I need to commit or, or, or not." Uh, um, if I don't know what I'm doing, I might be like, "What?" And, like, this extends to very complicated situations. And so what we started showing was this thing here, where we guess what the next action ought to be, uh, of the user, and surface it with, like, a single action. And what's cool about this is when I go into this, I immediately get into the agent flow. And so now I, uh, I'm asking, I'm, like, s- for, probably for the first time, I'm seeing, like, "Oh, wait a second, Warp can just, like, use an agent to fix my thing for me?" Right? "Is this cool?" [chuckles] And I'm like, "Okay." And now, even better, it's engaging me in a conversation here. So I'll be like, um, "Okay, wait a second, I just type English into this?" So I'll be like, uh, "Commit the changes." And I'm like, "Is this gonna work?" And then I type it, and-It just does it. And so something like this, where you catch someone in the moment, um, this was a really big unlock for us. This was probably our first sort of like, um, like inflection on the, on the agent side, was when we added this. Does this make sense, what I'm showing you?

    7. AGAakash Gupta43:28

       This is a huge unlock. I wanna just double-

    8. ZLZach Lloyd43:30

       [laughs]

    9. AGAakash Gupta43:30

       ... click on the learning for folks, right? Throw out-

    10. ZLZach Lloyd43:32

        Yeah

    11. AGAakash Gupta43:32

        ... your old guided tours, your, like, tool tabs, your gamified onboarding, right? When we're talking about agentic products, we're thinking about, how can we get them into that flow when it's gonna be useful for them, and surfacing it for them, right? And-

    12. ZLZach Lloyd43:46

        Yeah

    13. AGAakash Gupta43:47

        ... giving them that a-ha moment that, "Oh, it can do this." And it's not about shoving it in your face with some copy or something like that. It's actually while they're working, suggesting to them. So it's almost like you're brainstorming what they should use the agent for, and you're just, you're showing it to them.

    14. ZLZach Lloyd44:02

        That was a, yeah. And, like, another way of thinking about this is we were like, "You know, what's the, what's the version of autocomplete for agentic work?" You know? Um, because I, I do think one of the, the magical things about the first generation of, of AI products, um, is just, like, autocomplete is such a, a easy feature to use, because it's like, it's all upside. It doesn't slow you down, um, and it's easy to ignore. Um, but it's there if you want it, and it only takes, like, one keystroke to use it. And so we were like, "Okay, what's the kind of like..." You know, we could literally do autocomplete, and we do do autocomplete, but, like, what's the prediction of when a person is gonna get the most value out of an agent? And again, what's cool about this, so the thing that, that drives that prediction is, um, is intelligence. And so, you know, we are able to sort of understand, okay, this is an error that someone has, where it would probably be useful for us to suggest what a next step is for them to fix it, and that suggestion can educate them on how to, like, use the agent, and it will actually solve a real problem for them. And so, yeah, that's like, that feature, I think, is, like, kind of the, the coolest, you know, thing, and it really did move the needle. It, like, really upped people's, uh, engagement, um, with the agent. And, like, people will hit that when they're in, like, horrible state of frustration [laughs] and have been, like, you know, googling for fixes to things that they've been working on. Um, and they'll see that thing come up, and maybe they'll be like, "This thing looks like Clippy. Is this gonna actually work?" But the difference is, like, it works these days. And so, um, you know, it, it's been a really, really big unlock. So I would definitely encourage people to think of, like, what are the assistive things in the moment where you can get people to, you know, really adopt the, the agentic workflow in your tool, whatever that tool is.

11. 46:15 – 53:00

    How to choose right metrics for your Agent

    1. AGAakash Gupta46:15

       So as you were rolling out these agentic features, obviously-

    2. ZLZach Lloyd46:18

       Yeah

    3. AGAakash Gupta46:19

       ... you know, you measure any product feature towards some of your key north star metrics. We saw it [laughs] did eventually move revenue and users-

    4. ZLZach Lloyd46:27

       Yeah

    5. AGAakash Gupta46:27

       ... but you probably also moved, like, the amount of code and interaction they were having with the product. But what are the things specifically for agentic features that you need to measure that are different from your average product features?

    6. ZLZach Lloyd46:41

       Yeah. So I, I think the biggest one that we focus on is engagement. Um, and you could think of that in terms of, like, what, like, breadth of engagement or depth of engagement. The thing that's gonna lead, in general, to, I think, um, sort of paid conversion, uh, is, like, depth of engagement, meaning, like, is, um, the, the best sign that we have that someone is gonna become a paid user is that they are doing long conversations with the agent, long tasks with the agent on a relatively frequent basis. Um, you know, there's other things that we will look at too. I, I don't think that the product metrics are all that different. Like, we'll look at, like, how many lines of code is an agent writing for someone? Um, uh, and then we'll look at, like, you know, more, more qualitative signals as well, like NPS and, like, we will survey our users and that type of stuff. Um, you could look at retention, like retention and engagement. Um, retention's another big one, and we wanna see over time, on a cohort basis, that the retention of people trying and using the AI features is going up. And so we're getting that. That's pretty cool. I think we also wanna see the sort of, like, smile, meaning, like, people who might have tried the AI earlier, um, are coming back to it and, like, using it more, which means, like, again, like, the product itself is getting more valuable, and the user behavior, uh, that we care about is changing. So tho- those are the biggest ones.

    7. AGAakash Gupta48:20

       And, uh, there's a lot of talk about AI evals, and I think for agents, the evals are particularly important, 'cause you don't want it to go and suddenly delete their production database, which was, like, a big thing that everyone talked about-

    8. ZLZach Lloyd48:32

       Yeah

    9. AGAakash Gupta48:32

       ... with Replit or something like that. So how do you think about the eval testing suite, those offline metrics?

    10. ZLZach Lloyd48:38

        Yeah, so this is a, this is a huge deal. So we, we, um, you know, when we first brought our agent to market, uh, we did it really fast, and I think that was the right thing. It was like, "Let's see if there's demand for this," and, like, uh, "Let's put the agent in the app." We didn't, we didn't do evals even to start. We were just like, "Do people wanna work in this way? Does it help them?" Then, um, we were like, "Okay, w- people wanna work this way, uh, but it's very, very frustrating when it doesn't work." And so, uh, it's a new type of building stuff, where there's this-Element of nondeterminism in it, right? Which is, again, if you've spent most of your career building SaaS apps, that's just not the case. Like, you know, you, you build something, you bug bash it, you put it in front of testers, you can observe, like, the crash rate or whatever, but, like, the thing is coded, and it does what you want it to do. With AI-based features or agent-based features, like, you know, they, they kinda have a mind of their own. So yeah, if you wanna really improve it, you have to, you have to measure. And so we started investing in evals. I think there's different kinds of evals. For, um, for the coding space, there happened to be some, like, well-known public ones, like Swebench and TerminalBench. And we decided to invest in those 'cause A, A, they're-- it's like that's awesome. There's a pre-built set of things that we can see how good our agent is against them. And, um, you know, the performance of the agent, also, I, I think this is an important thing to make clear for people building, is not just the model. Like, the model's a, an input, um, and, like, you know, if you're using Claude or using OpenAI or Gemini, whatever, you do get different performance characteristics. But the actual performance of the agent is, is a function of the model plus the prompt plus the context plus, like, you know, you might use multiple models. You might have to do things where you, like, summarize or compact the context window or truncate context. And so there's actually a lot of, uh, engineering that goes into, like, getting the most out of the model. Um, but yeah, so we, we started working against these public benchmarks. That was really helpful for us, and, and it actually was helpful not just from, like, an improve our product standpoint, but it was helpful from a marketing standpoint. Like, we got to actually be, and are still are, the top, um, agent on TerminalBench, and we're a top five agent on Swebench. Uh, and that gives us a whole bunch of credibility. So we did the public evals, and then we've also invested in our own internal evals, where, you know, the way to think of these evals is, like, you present the agent with a task, you present it with, um, some sort of, like, acceptance criteria, so it's kinda like writing a test. And, um, you have to build a sorta harness where the agent can, like, try to do the task, and you can evaluate whether or not it was successful. Uh, and you're gonna need to build infrastructure for that, or there are, uh, there are companies out there that will provide this infrastructure as a service, and we've done some combination of each of these. Um, but then you do that, and then the other thing that you wanna do is, like, you should, I, I believe, have some evals that, that fail. Meaning, like, if you're passing 100% of your evals, you won't really know over time if you're improving. So it's a little bit unlike adding, like, regression tests or something like that, in the sense of, like, if you wanna measure your progress over time, you should be having evals that some of them fail. And then, um, in addition to doing, like, sort of these offline evals, you wanna do something where, if you can, you are looking at, um, you know, real user data in the wild, and looking, you know, at sort of like, you know, anonymized conversations or something like that, where you're trying to understand, are there common, um, patterns, uh, where people are failing using your agent? Are there common patterns where, uh, the agent's really successful? And you might wanna market that. And so, like, I think th- it's a big data feedback loop that you wanna put in place.

12. 53:00 – 59:24

    How to Actually make Money with AI Agents

    1. AGAakash Gupta53:00

       So as I previewed at the beginning, some people are making money with agents.

    2. ZLZach Lloyd53:04

       Yeah.

    3. AGAakash Gupta53:04

       You guys have figured out how to do it. Others are lost.

    4. ZLZach Lloyd53:07

       [laughs]

    5. AGAakash Gupta53:08

       So who's actually winning, and what business model innovations work for agentic products?

    6. ZLZach Lloyd53:13

       Yeah, great question. So [lip smack] let me think about how to answer this. So I think, like, you know, the, the first thing that you want is you want willingness to pay. If you don't have willingness to pay, you're gonna have a problem because these, um, you know, the LLMs cost a lot of money. [chuckles] So I would start with that. The second thing that you're gonna want is retention. Um, and so there, you know, we're in a time where I think there's a lot of, like, people kicking the tires on a lot of different, um, agentic apps. And so, you know, you, you wanna make sure you're keeping, uh, revenue you're getting and that you're, uh, you know, on a cohort basis, improving. The third thing is margins, and this is probably the hardest thing if you're at the app layer right now. It's, I think, notoriously hard for agentic coding companies, um, in the sense of, like, it's just a very expensive service to run. And what's particularly challenging, and you'll see this with Warp and with Claude and Cursor, and, like, everyone is trying to figure out how to price this stuff. So, like, the typical SaaS pricing mechanism of a fixed price per seat subscription, I think it just doesn't work that well [chuckles] with agents because, um, there-- people use these things in kind of a highly variable amount. You really want something where, um, as the usage grows, you make more money, not less. [chuckles] So that's, like, a problem with these fixed price subscriptions, which, by the way, Warp has. So we have these fixed price subscriptions. Uh, and it's like we're in this awkward position where it's like the more someone uses it, the more it costs us. And so you can end up being in a spot with your business model-Where you're trying to make money off of, like, the breakage, like the unused part of the subscription. And so this kinda stinks. [chuckles] So you wanna, you know, again, it's like we're, we're iterating on this. On the flip side, the consumer expectation is often a fixed price subscription, and if you, uh, if you do a pure usage-based thing, which is kind of the obvious solution to this, then you end up in this, um, you know, weird place where the consumer is kinda, like, always, like, watching their meter and, like, feeling like, "I don't wanna spend more money on this thing." And so this is, like, the art of it. And so right now, our kinda best guess on this is, like, a fixed price subscription plus an overages model that is more usage-based. Um, I don't know where it will totally land. Now, there are better ways to do this. Uh, if you are in a business where you can-- where you are, uh, less about, like, improving productivity, but you have a more measurable, um, outcome, uh, especially one where it's an outcome that the agent can do on its own, like resolving a customer service ticket, then you're in a very, very cool spot because you can do something closer to outcome-based pricing where you, you charge for the-- something that's much closer to the value that's being provided. In the coding world, it's just really hard to do that. Like, um, you know, you, you know, like lines of code created is a famously horrible proxy for how much value an engineer is creating. [chuckles]

    7. AGAakash Gupta57:06

       Yeah.

    8. ZLZach Lloyd57:06

       And so, like, it's really hard to, to measure, and so you end up, you know, charging based on, like, you know, how much people wanna use it, and so usage is closer. Um, but yeah, that's like, that's like the challenge of pricing. And then there's this other challenge which I, I think is really interesting, which is, like, where does the value accrue in the stack? And so, you know, you have LLM providers, um, [lips smack] who are-- So actually just going all the way down. You have apps that are paying LLM providers that are paying hyperscalers [chuckles] that are paying NVIDIA. [chuckles]

    9. AGAakash Gupta57:43

       Yeah.

    10. ZLZach Lloyd57:43

        It's kinda like how I would look at it. Uh, and so NVIDIA is doing really quite well. Uh, [chuckles] and, like, everyone takes some margin, and so the question is, like, who's gonna be able to capture the margin? And, like, I think it has to do with where is there going to be differentiated value. And, you know, right now in the coding space, there is differentiated value at the model layer. Like, I would say at this very moment in time, probably Anthropic has the strongest models, although GPT-5 is really quite good and, like, is competitive. Um, Gemini at the moment, like, they're gonna release something. I'm sure it'll be competitive. But the more that there's a ki-- uh, like a competitive dynamic at the model layer, the stronger the position of the app providers. The more that, um, an app provider-- Sorry. The more that a model, uh, provider has, like, uh, sort of like a real differentiated advantage in a particular vertical with their model, kinda like Anthropic encoding, uh, they sort of get something closer to, like, you know, pricing power, and that's a, that's a difficult spot to be in. So, you know, I, I've said this before publicly, I'll say it again. It's in our best interest that there's, like, a competitive, uh, model provider layer. Even if there would be open source models is really-- would, would com- that are competitive would completely change the dynamic. But even just having, like, something where it's competitive at the layer of, like, gCloud, AWS, and Azure are competitive, like there's not, like, one cloud hosting provider, that puts enough downward pressure that I think at the app level you can build something that has differentiated value.

13. 59:24 – 1:06:00

    Why Traditional SaaS Pricing Breaks

    1. AGAakash Gupta59:24

       Mm-hmm. I think the information released a crazy chart which looked at gross profit, and I don't think they were able to verify all the accounting, but-

    2. ZLZach Lloyd59:31

       Yeah

    3. AGAakash Gupta59:32

       ... at the high level, it was, like, Anthropic 50%, OpenAI 45%, and as you move more towards the app layer, like a Lovable, it almost gets down to, like, 20%. Is that grok? Do you think that those numbers are probably accurate?

    4. ZLZach Lloyd59:47

       I think that's-- Yeah, I think that's today's world, I believe that the model providers are taking m- more margin than the, um, than the apps built on top of them. But I don't think that that's necessarily a forever thing. I think, like, you know, there's, there's one other dynamic which is at play, which is that, um, [lips smack] over time, and I, I think it's something, like, in the last year, like, the cost per token at any given level of intelligence has gone down by, like, 100X. Like I, I, I'm-- It's either 10X or 100X, and I think it's 100X. And so it just depends. If, if that's the case, again, then there will be value at different layers. If someone's able to maintain, like, a durable lead at the model layer, then I think, um, you know, they'll, they'll-- they're gonna get a big portion of the profits. But I, I think it's more likely, just honestly, even putting my bias aside, that it turns more into, like, at the model layer it's all about scale, scale, scale, and, like, the margin is, like, slightly lower, and at the app layer it's, like, differentiated value, moat. Like, um, you know, do you have, you know, switching costs? Those types of things that, um, that get you lasting value. But I don't know. It's all-- It's changing really fast. Like, I think the que- like, the answer to this question, like, a year ago, I was going back and listening to some podcasts, was like, "App layer, app layer, app layer," and, like, "Models are commodity." And now it's like, I think it's more like model layer, but, but I, I, I, I would just focus on building something that people find useful, wanna pay real money for, um, and thatThe important thing is that there's margin somewhere right now. Like, like if there's no margin in any place in this whole business stack, I would say find a different thing to work on. Um, you know, and that would be like the Uber and Lyft situation, where I think the margins are sort of fundamentally limited by, like, well, what do you have to pay a driver per hour? And like, there's no more efficiency that you're getting out of that. Whereas for us, it's like, uh, you know, the actual... It's a, it's a, it's some combination of software and hardware, but fundamentally, it's like efficiency of compute, and that's a thing that historically has, like, gotten way better over time.

    5. AGAakash Gupta1:02:02

       Yeah. We've seen it just get way better. That cost per token at that particular intelligence level, that's the most telling.

    6. ZLZach Lloyd1:02:08

       Yeah.

    7. AGAakash Gupta1:02:09

       So where is all this heading? Like, what's your most contrarian take about the future of AI agents that other founders would disagree with?

    8. ZLZach Lloyd1:02:17

       Yeah. It's a great, [sighs] it's a great question. Um, I think there's three, there's three phases, and I'll speak to the coding world, if that's okay. Um, but I bet you this is true in other, in other domains as well. So the first phase was autocomplete, and like Cursor crushed it. Like, even before Cursor, there was, um, you know, there was Copilot, which was a good product, but frankly, I think it's... I love that a startup came along and was just like, [chuckles] "We're gonna do this way better." And like, basically like, in like literally, the difference between Cursor and Copilot is that Cursor's suggestions are faster and better, and it's crazy to me that they just, like, outcompeted Microsoft on that. Um, but that was, that was phase one. I think we're in phase two right now, which is what I would call interactive agents. And so these are agents where, um, a person behind a keyboard is saying, "Do this, do that, make this change for me, solve this, you know, debug this server crash for me." Um, and it's really like a human orchestrator. Um, and I think-- Actually, I think we're early in that phase. I don't think that that's like, if you survey most developers in the world, maybe they've tried to do this a little bit. I don't think it's their bread-and-butter workflow, but I think over the next year, um, the majority of development tasks are gonna start with a prompt, which is a pretty crazy change. Like, keep in mind, I've been doing this for like 20 years, and I've always-- For 19 of them, when I wanted to build something, I would just, like, open up my code editor and start typing, and now I'm just like, you saw me, I'm literally speaking to Warp, which is crazy. And then I think phase three is, is kinda like real automation, um, of some subset of simpler tasks. And again, I think you'll probably see this pattern in like, in, I don't know, I'm gonna say like legal tech. Like, it's probably right now, like the state-of-the-art legal thing is probably, like, someone using Harvey and like being like, "Summarize these documents for me," or, "Draft this contract for me," and there's a person driving it. But I think that the, the, the step that comes after is like, okay, I don't know, lawsuit was filed. Let me s- an agent's gonna start doing discovery or so- [chuckles] something like that. Or like, you know, uh, like I think that that will, that will happen. When that happens, I don't know what, um, I don't know what it means for knowledge work. Like, uh, I, I don't really believe the timing estimates. So like I listened to a podcast with a couple folks from Anthropic, um, and they were like, "There's not gonna be any knowledge workers in like, you know, three wor- three years" or something. I was like, "I don't think so." Uh, I don't, I don't agree with that because I think there's an underestimation of how difficult it is in certain industries to deploy technology. So like, like in healthcare or whatever, I don't think it's gonna be all agents. Um, but I think that you will see in some, um, you know, in some areas of knowledge work, specifically at startups and in non-regulated industries, you're gonna... it's gonna be s- super disruptive. And I guess that's not a super contrarian take, but like I, I, I, I don't think people get it. Like, you know, I live in like, um, um, for most of the year these days, I'm living in New Mexico, and I don't think, uh, most people in like my town know what, like, an AI agent is, and I don't think that's gonna last for that long. [chuckles]

14. 1:06:00 – 1:10:55

    AI Agents will change the way you work

    1. AGAakash Gupta1:06:00

       Yeah. It's actually crazy how low the adoption and awareness of AI agents is for people like you and I who are just living and breathing this.

    2. ZLZach Lloyd1:06:09

       It's... Yeah.

    3. AGAakash Gupta1:06:09

       I asked PMs, like, "How many of you are using an AI agent in your day-to-day work?" Which is a little different than developing AI agent products, which is what we're talking about today, but-

    4. ZLZach Lloyd1:06:18

       Yeah

    5. AGAakash Gupta1:06:19

       ...2% of PMs are using [chuckles] AI agents to improve their work.

    6. ZLZach Lloyd1:06:22

       Yeah. That's gonna change. I, I don't, I don't think... Uh, it s- it speaks to two things. So one, it speaks to like me and you are in some crazy tech bubble, where all we talk about all day long is AI agents, which is, [chuckles] you know, ooh, it makes sense given what we do. But, um, it also, I just think it speaks to, like, how early we are in the game of, like, this stuff truly being deployed. Um, but I don't know. Like, I'm, I'm, I'm not like a big, like, hype guy, but I, I am like, this is a, this is, like, a very fundamental foundational new technology that humanity has discovered and, like, it's gonna change our lives. Uh, even, even if the pro... Even if, like, we don't get whatever AGI or ASI very soon, it's like, it's already, like, the power of these models, uh, to do s- to do really useful stuff is very, very high.

    7. AGAakash Gupta1:07:19

       Yeah. I wasn't sure. Like, as a content trend, I had hopped on many different content trends over my content writing journey, whether it was crypto or-

    8. ZLZach Lloyd1:07:28

       Right

    9. AGAakash Gupta1:07:28

       ...NFTs or metaverse. After having gotten burned by all of those-

    10. ZLZach Lloyd1:07:32

        [chuckles]

    11. AGAakash Gupta1:07:33

        ...now AI is another one of those. But it's obviously proven itself not to be-

    12. ZLZach Lloyd1:07:37

        It's not

    13. AGAakash Gupta1:07:38

        ...a fad for years. And so-

    14. ZLZach Lloyd1:07:40

        Yeah

    15. AGAakash Gupta1:07:41

        A lot of PMs are now sold on this. They've watched, they've listened to us for an hour. For a PM who wants to skill up, like-

    16. ZLZach Lloyd1:07:48

        Yep

    17. AGAakash Gupta1:07:49

        ... they wanna build, like, a 90-day roadmap to get better at building agentic develop- agentic products, what should they do?

    18. ZLZach Lloyd1:07:56

        So first of all, they should, like, get hands-on in a tool. Um, I mean, obviously I'm biased towards Warp, and Warp is great for this, but, like, it doesn't have to be Warp. Um, you will be amazed what you can do by simply telling a tool to do it. And so a really cool way of doing this is to, like, you know, instead of writing a PRD for your next thing that you are, you know, product managing, or maybe there's a thing that you think the team should work on and people aren't even bought in, maybe build it or build, like, a simple version of it or a lo-fi version of it. Um, we've had this happen on our team a bunch, uh, with our designer, who's awesome. And, like, there have been times where we were like, we were like, "You know what? Building that thing is gonna be such a pain in the ass. We're not gonna build it right now." And this was, like, we wanted to build, like, a better, like, natural language classifier in Warp, which is, like, not a simple thing to build. And our designer, he knows, like, how to code, like, a little. Like, I, I, I don't think he'd be offended if I said that. He r- he's not, like, an expert coder, but he was like, "You know what?" I... He's, like, fearless. He's like, "I'm gonna sit there with Warp, and I'm gonna, like, do it, do it, do it until I have a prototype version of it that I can prove to the engineering team, um, can actually be better than what we currently have." And he did it, and it was crazy. [laughs] And so it's such a, like, empowering thing that I... Like, my first step would be, like, try and build something with the technology. The other reason to do that is, like, it will give you a feel for what is and isn't possible, which will inform the types of features that you are advocating for, um, for people to build. Meaning, like, you wanna build, like, a really, really good instinct, like, around, like, how good are these models, for instance. What can agents do? Like, can they actually, like, you know, organize your calendar, or are they gonna do some really stupid stuff? And, like, the way that you can get that feel is by using them in a very interactive, uh, context in your day-to-day. So highly recommend that.

    19. AGAakash Gupta1:10:15

        Couldn't endorse that advice any more. This has been a master class. Thank you so, so much, Zach.

    20. ZLZach Lloyd1:10:23

        Thank you for having me, Aakash. This is, uh, this was really, really awesome and really fun to come on. So thanks again.

    21. AGAakash Gupta1:10:30

        All right, everybody. Agentic AI, you have me saying it, is the number one trend right now that you need to learn as a product manager. Warp is proof. They're adding 1 million ARR every 10 days. We have made the playbooks public in this episode. Go act on this. Like and subscribe for more complete courses on AI product development, and we'll see you in the next one.

    22. ZLZach Lloyd1:10:53

        Amazing. Thank you.

15. 1:10:55 – 1:11:25

    Outro

    1. AGAakash Gupta1:10:55

       I really hope you guys enjoyed that episode. It would mean a ton to me and the team if you could please subscribe on YouTube, follow on Apple and Spotify podcasts, and leave a rating and review. Those ratings and reviews really help grow the show and help other people discover the show, and they help fund the production so that we can do bigger and better productions. Can't wait to share the next episode with you. Until then, see you later

Episode duration: 1:11:31