Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Aakash GuptaAakash Gupta

How to Talk to 100,000 Customers with AI (Ex-Bolt CPO)

Greg Greiner ran product at Bolt. Now he's co-founder and CEO of Woven, and in this episode he builds an AI research agent in Claude Code that emails every customer who drops off your funnel, then adds an eval loop live so the agent can…

Greg GreinerguestAakash Guptahost
Nov 13, 202658mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

  1. 0:00 – 1:36

    Intro

    1. GG

      So back when I was a CPO at Bolt, I would've been, you know, happy to talk to seven or eight customers a month. And now with AI, you have the opportunity to 1,000X or even more how many customers you're able to engage with on a regular basis.

    2. AG

      Meet Greg Greiner, former CPO at Bolt and head of product at Uber for Business, now CEO at AI company Woven.

    3. GG

      In the modern world, you should be using AI to create a scaled representation of yourself to talk to your customers for you. It should also come back with what it's learned and share insights with you on a regular basis. As PMs, we really need to get close to customers, and this is how we do it. Essentially, it's scaled research with AI.

    4. AG

      How has AI changed what PMs need to be able to do?

    5. GG

      I think it's now possible to actually have real back-and-forth conversations with thousands or even hundreds of thousands of customers every month in a way that would previously require an army of researchers, an army of support reps.

    6. AG

      You cooked it up in just 30 minutes. This is something I haven't seen somebody build before. Okay, what is the groundedness? What is the true data source? Before we get into today's show, please take a second to check that you're subscribed on YouTube and following on Apple and Spotify podcasts. If you want access to all of my favorite AI tools, I've gotten them to give you an entire year of their paid plans. Check out bundle.aakashg.com for an entire year of Bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arise, and Mobbin. And now, into today's show.

  2. 1:36 – 6:32

    What AI research on every drop-off can tell you

    1. AG

      Engineering is getting faster with AI. Is product management really getting faster? One of the ways I would argue it is, is by helping you talk to more and more of your customers. Now, of course, you've been using Claude to help you synthesize transcripts, but have you been using it to drive automated proactive research with every single one of your customers? How might you actually do so? If you really want to get to that next level of customer research with AI, today's episode is for you. Greg, welcome to the podcast.

    2. GG

      Thanks for having me.

    3. AG

      Greg, you told me something pretty interesting. When you were chief product officer of Bolt, you would be happy to talk to five to 10 customers a month. [chuckles] That has really changed with AI. What is this new level of AI research report that you can build?

    4. GG

      Yeah. I, I think it's now possible to actually have real back-and-forth conversations with thousands or even hundreds of thousands of customers every month in a way that would previously require, you know, an army of researchers, an army of, you know, support reps, um, to, to facilitate all of those conversations. Uh, and when you are able to have all of those conversations at scale, you're able to produce really interesting insights. Uh, so this is an example for the company ancestry.com of, of what one of these reports might look like if you're able to have these scaled conversations with customers. And so one of the first things that we always report on is obviously the impact in the customer journey from a business metric standpoint. So we found that, you know, having this type of personal conversation with customers doesn't only capture insight, but it also allows you to convert a broader set of this audience. So in this case, this is people who've abandoned the process of purchasing a DNA kit with ancestry.com. And so beyond that conversion insight, uh, you're also learning, you know, for these people who specifically dropped off in that funnel, what are their biggest concerns, right? What are they bringing up most frequently? What is actually driving them to abandon? And then in addition to that, when you layer on the conversion data about what people actually do after these conversations, whether they've finished their purchase or not, you can double-click a layer deeper into what are, uh, the, the relative conversion rates of these buckets. So not only like how often are they specifically happening, but when they do happen, how do these people behave, and how does that compare against the baseline of all the people who engage in conversations? And so you can sort of see whether something is, is a, an issue or a hesitation or a concern that is easily addressed with just messaging, right? The AI responding back to the customer and sharing, you know, specifically answering the question around shipping and delivery time. That's something where this can really just be solved by updating your messaging, maybe more proactively talking about shipping and delivery time, versus something like subscription confusion, that even when the AI has the chance to respond to the customer and address that concern, these people are still not converting at a high clip. And so that opens up an, a broader opportunity for you to dive into that problem and try and address it in a more holistic way with your product.

    5. AG

      Fascinating. So these conversation counts are so super high.

    6. GG

      Yeah.

    7. AG

      How are these conversations generated?

    8. GG

      So these conversations happen, you know, when someone drops off in that flow after they've, you know, entered their email address, let's say, or their phone number. The AI is reaching out over that channel, and we're getting a large percentage of those people to respond and actually engage in a back-and-forth conversation. And so that's where we get to the scale of we have customers doing, you know, tens of thousands of these conversations, hundreds of thousands of these conversations every month, and generating this, like, more scaled insight into the specific audience at the specific point in the journey.

    9. AG

      So is it al- is it this basically like the evolution of in-app surveys plus AI?

    10. GG

      Yeah. I, I think that's a good interpretation, and I think what this allows is not just the survey from the insight perspective, but you can dive a layer deeper, right? When someone says, "Oh," like, "I- I'm concerned about," right, this example on the left here, like, an unexpected relative popping up and that being awkward, right? You get the chance to not just, like, uh, take that in as a piece of data, but you get to double-click on it and ask follow-up questions, attempt to resolve those concerns, and see how the customer reacts to that, which gives you a much richer sense of how these different, uh, things are actually impacting your customers more directly.

  3. 6:32 – 7:00

    How to build your own research agent in Claude Code

    1. AG

      Awesome. So how do I build this?

    2. GG

      Yeah. So I, I think, you know, for a lot of people, it's not gonna make sense to, you know, buy a dedicated product like Woven to, to accomplish this outreach. And so the awesome thing is that with existing tools, you can build something very similar to this, um, using something like Claude Code. And so, um, you know, in about 30 minutes, uh, before the podcast, I put together a quick example of this. And how it works is-

  4. 7:00 – 8:52

    Ads

    1. AG

      Most roadmaps are out of date after the quarterly review. A PM spends a week rebuilding it in slides, leadership looks at it once, and two weeks later, half of it is moved while the deck stays in someone's inbox still saying the old thing. The other half of the problem shows up when an idea gets committed. The customer insights and the reasoning for picking it live in one place, the tickets engineering works from live in Jira, and the only thing connecting them is a PM copying and pasting. I've been using Jira Product Discovery for both of these, and each one takes about a minute, so let me just show you. This is my discovery project. Every idea has the insights behind it attached, things like customer quotes, support tickets, and usage data, along with the fields my team uses to prioritize. To share it with leadership, I publish it. They get a link to a read-only version. They don't need a license to open it, and they only see the fields I choose to share. And because the link is live, when priorities shift or something moves next month, the roadmap they already have updates too. No more roadmap_V7_final. So that's the living roadmap. The second piece is connecting discovery to delivery. Say this idea is committed for next quarter. I open it, go to the Delivery tab, and create the delivery ticket from right here. It lands as an epic in the engineering team's Jira project already linked back to the idea it came from. As engineering moves that work through their sprints, this progress bar fills in. Add the delivery progress field to the roadmap view, and the link leadership already has shows where each item actually stands, so nobody has to ping me or set up a status meeting to find out. That closes the loop. The reasoning stays with the idea, the roadmap is a live view of it, and the work engineering ships stays tied to why it got built in the first place. If your team already runs Discovery in Jira, be sure to try out Jira Product Discovery.

  5. 8:52 – 12:29

    Demo: an agent that emails customers who drop off

    1. AG

      Link is in the show notes.

    2. GG

      It takes in a source of customer data. So for the demo, it's just a, a list of customers on a CSV, right? So, um, people, their drop-off points. You know, this is f- for people who've dropped off in a trial, data about their usage, et cetera, that allows the AI to personalize the outreach and, and the conversation. You can also use any form of, you know, CDP or a data warehouse like Snowflake or Segment to power this. And then it has some basic instructions on how to approach the conversation, how to format the email, um, in the Claude.md file. So this is all relatively straightforward stuff, but it's important, um, to ensure that you're, you know, still making this feel personal. It still feels like an email that you would draft back to the customer directly. And then I filled in some basic stuff. This is for a, you know, made-up company called Cadence that does AI, um, fitness training. Um, and so some information about different escalation processes, different policies that the company has around things that the agent can do or not do, and then the specific programs. But for a lot of companies, this data already exists, and so you already have pretty rich markdown files about your company that you can point the agent at. Um, and even if you don't, um, you can use Claude pretty easily, point it at your website, and it'll generate a bunch of this stuff on the fly. And so it uses all that information to, um, personalize outreach to this customer audience. Um, and so I will trigger that really quick in my Claude session. So there's a skill that just does the, the outreach sending, but you can also either schedule this or have it trigger based on new entries entering this, um, CSV or from the database. Um, and so what it's doing behind the scenes, uh, when it does this outreach is it's going through that list of customers. It's identifying the ones it hasn't reached out to yet. It's, um, taking that metadata about the customer, personalizing a specific outreach, um, that is relevant to that customer and their situation and the goals of the outreach, um, and then sending that directly to the customer. And so I can show a couple examples of what that looks like. So this is one I did last night right before the podcast, um, reaching out to someone, right, who has started using this app, but, um, you know, hasn't fully converted from their trial, you know, asking specifically about, um, their experience with the program. What were their thoughts about, you know, how intense it was or what the level was? Um, you have the customer respond. The agent is able to get back right in a couple of minutes. Um, you're able to continue that conversation, continue asking follow-ups to gather more information. Um, and one of the things that we found with Woven is A lot of the value of this personal outreach exists when you're able to respond quickly. Um, so we've added and tested adding delays to the response mechanism, and you see the reply rate drop precipitously, right? So people are likely no longer in their inbox, you know, likely no longer in the workflow of doing their email, and so you lose them at that point. Um, and the conversion rate also drops alongside that. And so that's where AI can come in and enable you to, you know, respond really quickly, really accurately to these customers at scale, and to get that benefit of these long-running conversations where you're really extracting a lot of insight.

  6. 12:29 – 16:03

    Why email beats in-app chat, and 20% to 40% reply rates

    1. AG

      So why over email? Why not just within the app, I guess? Wouldn't that be a little bit smoother?

    2. GG

      Yeah. So I, I think, um, in the app also definitely makes sense, and we work with customers and recommend adding in-app chat as well. But email has a few benefits. One, this abandoned audience, right? They're not in your app anymore. [chuckles] Um, and you also, uh, need, need to be careful when you're doing things in app about not distracting from the core, um, goal of someone being in your app, right? You don't know that they're going to abandon yet, and so you don't wanna actually have this, like, research-oriented chat end up-

    3. AG

      Yeah

    4. GG

      ... with, uh, it, it basically hurting conversion instead of helping. The other benefit is it's really easy to roll out and scale. It's a channel that all of your customers already use on a daily basis, and so you're able to be much more proactive and aggressive. And so we get, especially with these research campaigns, um, you know, 20, 30, 40% of these people responding, which when you think-

    5. AG

      Whoa

    6. GG

      ... about how many people would interact with an AI chat that exists on a, on a site, right, it's, it's much, much lower. Um, and so you get, uh, a broader, uh, ability to capture that audience.

    7. AG

      And can we just maybe break down, like, how you drive... Like, what's the difference between driving 20% response and 40% response?

    8. GG

      So a lot of times it is around the, um, type of campaign that you're running. And so I think when you are doing something that's very research oriented, and if you're doing something that's a little bit later in the funnel, meaning people have already shown some intent, right? They're, like, in this case, like, already using the app, have, have had experiences with it, much more likely to respond and much more likely to get up to that higher range of responses. Um, we just did a, um, an experiment with a company in the, you know, fitness space, and they saw about a 35% for this exact use case of trial abandon. And then as you go down, it's often, you know, lower in or earlier in the funnel, right? Maybe someone just entered an email but didn't actually finish an onboarding process. Um, and so there's less intent. But there's also matters, you know, how well you do this initial outreach, how personal it is, how relevant it is to them that is gonna drive whether people respond. In a similar way to, you know, in B2B sales, you know, cold email outreach, right? The better you make this email, the more likely it is that this person's gonna be willing to take the time to respond.

    9. AG

      So if you had to give, like, your average PM at a startup a template for this email, what would it look like? Would it be this kind of plain text sort of version so that it seems more human?

    10. GG

      Yeah. I, I think that's the right approach. Um, and, you know, asking simple, direct questions that are really relevant to where the user is at at this point in the journey allow you to, you know, open the door to them responding, sharing some information, and then you can expand on that. Um, I think, uh, mistakes people make is, you know, too much content, too many... You're a- you're trying to ask too many questions at once, trying to essentially, like, recreate your survey maybe in one email, um, when if you take it a little bit slower, one, you can, you know, let the customer lead you towards what they care about and what they wanna share insights about. But also, you can often get to all of those questions through the back and forth of the conversation at a much higher rate than if you're trying to, you know, stuff them all into one, one specific outreach.

    11. AG

      Mm.

  7. 16:03 – 24:15

    The four emails to send across the customer journey

    1. AG

      Mm. If you had to think about, like, what are the right set of emails for someone to create, these are, like, the four points along a typical-

    2. GG

      Yeah

    3. AG

      ... journey, how would you map those out?

    4. GG

      Yeah. I, I think, um, you know, if you take this example of this fitness app that has a trial, right? I think there's one at the very beginning, right? There's gonna be people who even start that sign-up process, like I mentioned, enter an email address, never actually, you know, finish creating their account, finish the onboarding process, select a program. And so that's gonna be the widest part of your funnel. It's gonna be the lowest response rate but also the highest volume. And so I think that's one that makes a ton of sense. It can also inform the rest of your journey downstream of that. I think the second touch point is it makes sense to do it somewhere in this trial and potentially, you know, multiple times within this trial where you're getting people set up for success, and then you're checking in on that success to see how they're proceeding through the product. And then I think, you know, when we're talking about onboarding specifically, the last one that makes sense is obviously a lot of the people in these trial situations are, you know, not converting, right? They're not going beyond that trial. And so I think it always makes sense to reach out to that group as well. And one of the things that we've seen make sense that I think is really easy to do, you know, as a PM as well, is at that point offer some form of incentive just for sharing insights, right? You, you don't have a lot to lose at that point, and so saying, "Hey, if you share, you know, things that you think we could do better for your next trial, I'll give you 20% off, um, you know, your subscription for the next few months." Um, and so what we've seen that do is it massively increases reply rate, um, but it also increases reply rate for the audience that's already on the fence, right? And so, like, you're not gonna reply to that email for that incentive if you weren't already considering it. And so you get a, a captive audience who you can both address their concerns, right? Like, hopefully make them feel good about u- about purchasing the app, while also pairing that with, um, a, a promotion that you were probably gonna send anyways, um, to get them, you know, to take that next step and, and come back and convert through the funnel.

    5. AG

      So I guess that's the onboarding stage. What about further? If we go into-

    6. GG

      Yeah

    7. AG

      ... once they're engaging in the core app, once they're retaining-

    8. GG

      Yeah

    9. AG

      ... once they're lapsing. What are the other points along the life cycle that this makes sense?

    10. GG

      I think when you're engaging the core app and successfully engaging the core app, you know, that's where the in-app experience and chat makes the most sense, right? That's where you should be capturing people in the moment in that experience. They're, you know, using this product on a frequent basis. I think where, you know, using these more traditional channels can make more sense is, again, at these points of drop-off or abandonment. So it could be an active user who has explored a, a f- a new feature or a, you know, an upsell feature for the first time but didn't actually convert. And I think that's a really unique opportunity. Um, or it could be someone who was, as you mentioned, like, previously active but has decreased their usage in the last few weeks. Um, or it could be someone who, um, obviously, like canceled their subscription, right? And so that's another opportunity similar to the trial, where every, you know, incremental conversion you save there is inc- is, is additive. But also, those are some of the most interesting insights you can find, are from this audience of people who have decided, you know, this isn't for them and, and the value wasn't good enough.

    11. AG

      Okay. Makes sense. And operationally, I guess, are these through, like, your transactional emailing system? I guess, like, a lot of times, marketing ops will own that.

    12. GG

      Yeah. So I think for, you know, these tests, you can do this, uh... I did this with a standard Gmail account, and so that limits, obviously, the scale of the outreach that you can do based on what's reasonable to do from Gmail. Um, but it does allow you to do it, you know, completely by yourself without using an existing or, or, you know, system for, for marketing, right? Like-

    13. AG

      Yeah

    14. GG

      ... this is all just happening from this greg@cadencedemo.com. These are the outreach I just sent earlier. Um, and so I think that's very possible if your, you know, audience isn't massive. And then I think, yeah, if you're, if you're dealing with a larger audience, you would likely need to partner with, um, your marketing team to use an existing tool. Or it's really trivial to set up, you know, uh, Postmark or another type of tool specifically for this use case. And, you know, honestly, relatively common, um, for research-oriented use cases to already be sending, you know, from a different service, even if you're using, you know, the legacy systems like Medallia and Qualtrics. All of those are not, you know, are not sent from the marketing team, right? They're sent from those systems. So, um-

    15. AG

      So you could stand up your own sort of, like, product research-

    16. GG

      Exactly

    17. AG

      ... email system.

    18. GG

      Exactly.

    19. AG

      And that's what's common best practice. Okay.

    20. GG

      Yeah.

    21. AG

      I didn't know that.

    22. GG

      Yeah.

    23. AG

      Very cool. So what I'm taking away from this is email is this super powerful channel. You should set up this stack where you have the ability to email your customers, and you can make it a lot more intelligent by emailing them at the right time via the right triggers based on their life cycle. Typical things like trial, didn't sign up, lapsed, churned finally. Then on top of that, you can layer in instant AI responses in your voice to go deeper on the conversation, go deeper on the insights that you obtain. At the same time, potentially address concerns that they have to improve the conversion rate at that step of the funnel.

    24. GG

      That's right. And I, I think for an individual PM, um, you know, I think this type of thing can be very powerful as you're trying to both better understand an audience for a sp- specific product or feature that you own, and then also, you, you wanna drive adoption of it, right? You want people to be using it. Um, you wanna be able to follow up with those people. Um, and so you also create this audience of engaged people that you can come back to these threads, right? Even if it's, you know, manually or, you know, you can have Claude take a pass through your inbox and identify interesting folks and say, "Hey, I'm launching a new feature. I wanna see everyone who mentioned something related to this feature, and I wanna reach out to them, you know, either with a follow-up question or with a, you know, invite link to jump on a call to talk through this, um, you know, feature that was developed based on their feedback."

    25. AG

      So we started with the output report. We showed you what the outcomes look like as email. You cooked it up in just 30 minutes. Can you at least get us started, show us how you would use Claude Code? What prompts that a typical PM might be using to get the scaffolding for this setup?

    26. GG

      Yeah. I, I think, um, pretty straightforward. I, um, mainly just gave Claude the, the context of trying to set up a specific outreach program from my inbox, um, using the Gmail MCP. Um, and so similar if you were gonna use, let's say, I put the Postmark MCP and, um, pointed it at the source of the customer data. So I think that's one of the main things that you need to figure out, is where are you gonna be getting th- this trigger data from. And then, um, gave it instructions on, "Hey, these are the, the, the context of the, the conversations I wanna start. I wanna be reaching out to people who are 10 days into their trial." Um, and then, uh, you know, let it essentially run with that context. And, you know, the first pass needed some tweaking, right? There were things like the threading was broken on the replies, right? They didn't get actually threaded back into the email or, um, you know, some of the, the language was, was inappropriate. And so there's some optimization that you can do just by having it run through tests like the ones I'm showing here. Um, but I think even with that- Base instruction, you end up with, um, something that's working, like I said, in, you know, 30, 40 minutes that you can start testing with real customers.

  8. 24:15 – 31:23

    Adding an eval loop to the agent, live

    1. AG

      So there's two things I think is really, really interesting here. The first is that as soon as we're going to talk to the customer, we probably need to think about what are our online evals to measure that it's actually pulling up-to-date information from our real help center, that it's helping drive objections down in the way our sales team want it, that it's maintaining our corporate brand of voice. And then there's this component of automation, of how do we create this as a system that can quickly respond. So can you walk us through how you might build those?

    2. GG

      For sure. Yeah. I, I think, um, for this demo, we already have this classification process, but one of the things that we can, you know, showcase live is the next step of this, which would be adding evals as a separate step to this process that, um, run before the actual reply gets automatically sent to the customer. And so if I go over to Claude Code, I'll zoom in a little bit here. Um, you can essentially just ask Claude and, and, you know, explain what you're trying to, to output, and if you've already put a little bit of work into this, it should, um, be able to give you a, a pretty good approximation of this. So we can say, "We now wanna want to add, um, an eval step after classification, um, for responses that are queued to be auto-sent. Um, focus that eval step on groundedness in the documentation. If the eval fails, rerun the generation process and save the new generation as a draft for review." So I think this allows, um, you to do a couple things. One, it sets up the process for evaling, and then I think you can then, um, build additional evals beyond just groundedness on top of this. And then two, it starts building this loop that I think is really important of, um, when an eval fails, what happens. So, um, the eval should not just, you know, stop the whole process. Ideally, you're using the information from that eval failure to have the agent take another pass so that you can still, even in these cases of eval failures, get to automation that you feel confident in.

    3. AG

      This is something I haven't seen somebody build before. So basically, you're building the evals live into the product. It's-- and here we're focused on groundedness. What Claude is doing is it's building this system of judging, okay, what is the groundedness? What is the true data source? What is the LLM judge? And then, like you said, it's creating this loop. If the LLM judge views it as non-grounded, update to the grounded version, and then actually send it.

    4. GG

      That's right. And I think, you know, what we've seen on our end is that, you know, for the vast majority of cases where these evals are failing, you can pretty quickly get to, uh, a positive response, um, that passes the evals and, and the guardrails. And I think one thing that's important as you're building this is you don't also wanna be necessarily overfitting, um, to, okay, it's just taking this exact eval, and it's, like, following it to a T. You ideally want to use the eval failure as input to another full run of your process of generating the response, um, so that it's, uh, you know, still going through that full logic loop of understanding how to respond and taking the eval in just as input versus, like, here's the exact instructions, right? You wanna update things.

    5. AG

      This is fascinating. I can't believe I haven't, like, heard about this before. But basically, you're building quality into your product so that there's no non-continuity, that it doesn't just stop. And at the same time, you're probably improving your performance on your online eval. So if you had a groundedness metric, probably if you launched this loop, you would see a bump up in that groundedness metric because it's getting the feedback live.

    6. GG

      Yeah. Exactly. And that's, I think, one of the benefits, uh, we talked about email earlier. One of the benefits of email is, um, there's not an expectation of instantaneous responses. Um, right? It's actually a little bit weird if you're responding, like, within seconds.

    7. AG

      [chuckles]

    8. GG

      Um, and so by having a little bit more time, you can run these loops, um, in a way that you can't if it was a chat-based experience, because it would just be, like, someone sitting there waiting for, like, three minutes for the chat to respond back. Um, and so that's what allows for this process to exist and still, like, deliver a good customer experience because there's not a, a need to actually, you know, respond, like, within, whatever, 300 milliseconds.

    9. AG

      So what's pretty fascinating is Claude had no clarifying questions, so there's probably some risk that it's gonna go off into some direction [chuckles] that we disagree. So as it's finishing up its response here, what are we gonna look for?

    10. GG

      Yeah. So I, I think you wanna understand, um, sp- specifically, I think, I think what changes that it made and how that's impacting your process. So looking at, um, where it's inserting updates to, like, this process of processing the replies, um, how that's, you know, manifesting itself into the, the summary that it's writing here, and making sure that the summary is aligned with, you know, what you actually intended the output to be, right? So, right, the, the evals are only gonna run in auto-send mode. That makes sense, right? If it's not on auto-send, we're already going to draft immediately. Um, the groundedness eval will check generated replies cr- claims against- Um, the underlying documentation. I think that's also accurate. And then on failure, regenerate once with the specific issue corrected, and then save to a Gmail draft, which was what I instructed it to do. I think per our conversation, the, the next stage of this is probably, um, updating, um, this to, you know, retry multiple times potentially and get to that approved, um, response that you can auto-generate. And then I think, you know, the power of AI too, right, is it's doing things that it didn't, I didn't even ask for, right? As a part of this, um, workflow, I have, uh, Slack notifications like this one from the one I sent earlier that get sent when, you know, an inbound message comes in and it's processed. Um, and it automatically added, you know, this failure emoji for the case where the eval fails, um, which, you know, didn't even have to explain that that was, that would be valuable.

  9. 31:23 – 35:46

    What Woven adds, and when to talk to customers yourself

    1. AG

      Okay. Fascinating. So we just showed you how you could potentially be engaging with almost all of your lapsed customers, downloading information from them. We showed you how you can vibe code this as a system built in Claude Code with proper eval so that you have confidence when you're putting it in front of your customers. Not everyone's gonna wanna build this themselves, so what's the experience like if they choose Woven?

    2. GG

      For sure. So I think a lot of this complexity is, you know, managed for the customers, um, who work with us. And so the process looks largely the same. It's just intended to, one, be at much greater scale, and two, have a bit more of a conversion orientation, like we talked about the, you know, the more research-oriented campaign versus the conversion. And so, you know, going back to that ancestry.com example, um, this is an example again of someone who abandoned their DNA kit. These outreaches get sent at scale based on integrations with your marketing automation platform or CDP. So we handle that kind of data, um, pass through. Customers respond, and then the AI gets back to them. And then as a part of setting up one of these engagement points, we generate a bunch of simulated examples, which allows you to essentially give feedback to improve the responses directly. And so this is how, without having a bunch of back and forth with Claude, uh, without having to build all of this yourself, you can quickly get to confidence and also see and visualize, "Hey, here are the changes that, um, the agent is specifically, um, making based on my feedback," right? So it created a guardrail, um, it created a playbook, it updated the agent context. Um, and so you can review all these things and gain confidence, but then also see the output and how it's been improved. Um, and then on top of that, right, we talked about evals and guardrails. That's something that, you know, we've invested a lot in defining really strong guardrails globally that, you know, handle things like groundedness, handle things like, you know, the formatting of the email. And then in addition to that, as a part of this training process, we're setting up custom guardrails that are based on the feedback you share, based on what matters most in terms of, um, ensuring quality. And so those are then run on every response, and like we discussed, are rerun essentially until we get to a passing response. Um, and so that's, uh, you know, one of the things that makes this powerful is, is our customers. Like, we have customers who have hundreds of these engagement points live with us right now. Um-

    3. AG

      Wow

    4. GG

      ... and it takes them like five to 10 minutes to set a new one up once they've gone through this process a couple of times. Um, and so, you know, y- you can kind of hit on, we talked about all the different points in the journey. Start with something straightforward like this, you know, cart recovery use case. You can start hitting on, you know, these conversational campaigns basically everywhere where you have questions about customers, where you wanna be influencing customer behavior.

    5. AG

      When is this not the right choice? When should a PM be u- by hand emailing, looking up customers, making sure that they talk to the customer instead of the AI talking to the customer?

    6. GG

      I think it, it's, um, dependent on scale and I think also where you are on the, like, product market fit journey, I would say as well. So I think in cases of lower scale, um, right, there's a lot of value in having those more direct conversations. There's also a lot of value in having conversations that are not, you know, text-based, right? Where you're actually meeting with something, someone, building a relationship, really understanding them at a deeper level. And I think that's also true, you know, when you're in the earlier stages of, of finding product market fit. Um, you know, you can't, uh, you know, fully automate that process of discovery, right? The AI agent is never gonna have the same amount of context that you have as the PM of this, you know, early stage product. And so, you know, this is for a little bit later in that product market fit journey where you already have a sense that you're creating value and you just wanna maximize that value.

  10. 35:46 – 40:28

    The full AI research agenda for a PM

    1. AG

      If you broaden it out, you're still at Bolt for a second.

    2. GG

      Yep.

    3. AG

      You now have this new capability. They've built out this system. What is the broader research agenda advice you're giving to a new PM on the team? What does, what does all of the AI-enabled research activities, not just this one, but all of the AI-enabled research activities they should be engaging in look like?

    4. GG

      Yeah. I, I think, you know, like you said, this is one piece of a broader AI research strategy. I think this, you know, makes sense for a constant drumbeat of, um, research, but I think there's opportunities to scale the other types of research as well, uh, in particular customer interviews. Particularly customer interviews. So, um, I think there are tools like Listen Labs and other types of products that enable you to automate these more, you know, voice-based back-and-forth research conversations. And so again, I, I wouldn't necessarily recommend those for an earlier stage company, but, you know, by the time I w- was, was kind of in the later stages of my career at Bolt, um, right, those are really powerful opportunities where, you know, we had a lot of B2B customers, but also millions and millions of shoppers on our network from a B2C standpoint. And so you have an opportunity to use those automated research interviews to dive a layer deeper, and potentially use these email conversations as an entry point to find the users who are-- have the most interesting insights to share. And then using that next level of, like, actually an, an AI-facilitated voice back-and-forth to, um, go deep with those people and get, like, a much richer set of feedback and context from them.

    5. AG

      So maybe you can react to this and see if this is about right. Because what I don't wanna do is I want people to see this episode and then all of a sudden purely pursue AI-enabled research.

    6. GG

      Yeah.

    7. AG

      So is this accurate that, like, a typical PM should still be regularly chatting with customers on Zoom calls, over text message, over Slack, I would say multiple times a week if possible. So I'd-- by multiple, I basically mean more than two times a week. Like, at least you're having a Slack chat or you're texting or you're hopping on a call or, of course, the best thing, you're literally watching them use your product in their context. I still think a PM should probably be doing that once or twice a week. But in addition to that, they should be setting up these AI-enabled research programs. Like you said, they should be setting up synthetic conversations with an AI avatar. They should be setting up email-based chats on key levels of the journey, like we demonstrated today. They should probably also be setting up some sort of platform that's analyzing all of their sales calls, all of their social messages. So they probably have at least these three AI-enabled, plus the regular discovery, which maybe is part of, like, a continuous discovery habit loop. So there's basically, like, four sort of key customer research activities a PM should be pursuing. Is that right?

    8. GG

      Yeah. I, I, I think that's right, right. I, I think, um, what these more automated workflows can enable is making those, um, true moments of, you know, interaction and relationship building much more powerful, right? Because you'll be coming into those conversations, uh, much more educated on your user, on, you know, what the foundation is, where there's potentially, like, interesting things to poke at and interesting things to dive deeper on. And so those don't replace those real moments of, you know, human-to-human interaction where you can really dive deep and understand someone. They just make them more powerful. You spend-- you waste less time in those conversations covering things that you should probably already understand. Um, and that's one of, one of the other things that we think is powerful about the more steady drumbeat process is, you know, when we're generating these research reports, I showed the example of this, you know, deck-based version, but we're also generating, you know, more true report-based versions, and we're sending those over Slack. And so that gives an opportunity where this, um, research is really living where people are working, and they get that stri- steady drumbeat of what are our customers saying, what are they actually doing, you know, after they s- they share this, this type of feedback. Um, and so that can then inform that more in-depth research that you do on top of this.

    9. AG

      So that is just a masterclass in what the state of first-class

  11. 40:28 – 45:45

    Why one PM can now run a team that ships like 25 engineers

    1. AG

      discovery and research looks like. You had some really hot takes on product management in general, and so I wanted to shift there.

    2. GG

      Sounds good.

    3. AG

      How has AI changed what PMs need to be able to do?

    4. GG

      Yeah. I think, uh, I've been reflecting on this recently, and one of the interesting, uh, you know, things that I felt at Uber was that actually the ratio of PMs to engineers was, was often way too high, right? You had teams of, you know, one PM and three or four engineers. And so you had these cases where, you know, PMs were looking for things to do. And I think at, you know, my, my greatest scale at Uber before I built a team, I was at about one to 25 engineers. And I felt that was actually one of the most productive times, um, for myself and the team, um, because I-- there was, you know, no room for anything but, you know, developing product and, and creating value and, and moving quickly. Um, no room for, you know, processes and all of these different things that PMs can get caught up in. Um, and so I think one of the things that AI enables is that, um, you can actually go back to that, you know, Uber-style ratio and have a PM and three or four engineers, but still, you know, create the output of that 25-engineer team. And so what that means is, you know, PMs will need to be shifting their mindset to be able to handle that level of scale, and that looks like being able to, you know, context switch very quickly, being able to manage a number of roles beyond the traditional PM job description, right? Like, we didn't have product marketing support. We didn't have all the design support that we needed. And so, you know, the PM becomes that person who needs to fill in the gaps when you have so many people producing. Um, and you need to be able to stay ahead of that level of output ... in a way that, um, still feels cohesive, right? It still feels like you're building something that's greater than the sum of its parts, which can be particularly challenging, you know, when you have 25 engineers or you have an AI-enabled team of four, four or five engineers, because the output is so fast and you're just trying to, like, hold on for dear life and keep up. And so that's, like, one of the biggest challenges facing modern PMs is being able to stay ahead, being able to continue to have these, you know, features that you're launching be, um, cohesive and, and additive and not just, "Hey, we're shipping everything a customer requests, and we're doing it as, as quickly as we can."

    5. AG

      Can you break that down in more detail for me? What are the processes that PMs used to get caught up in? How do you say no to those processes? A lot of times when I was working at a bigger company, I almost felt trapped by the process. I could recognize it in my time, but I didn't know how to get out of it.

    6. GG

      Yeah. I, I think often it's meetings, right? So often it's, you know, you have to get approvals, you have to, you know, jump through a lot of hoops. You have to, um, get, like, a bunch of stakeholders aligned. Um, you have to, you know, check some boxes for some planning processes at the corporate level. Um, and so obviously, y- you know, when you have this, like, torrent of productivity coming at you from the engineering s- standpoint, um, you have to get creative, right? You have to be able to, um, you know, push back on the meetings that you know are just, you know, busy work and cruft. And then for the others, you need to find ways, and I think AI enables this in a much, much more scalable way, to spend less time on those and, um, you know, automate more of that process of getting ready for and, and completing those meetings. And I think, you know, what PMs could get caught up in, in, in the, you know, previous world is that all of that cruft, all of those processes, all of those meetings were the actual job, um, and not actually, like, building product and talking to customers and releasing product. It was, "Hey, I'm gonna jump through the corporate hoops. I'm going to, you know, uh, check all the boxes on this playbook for how PMs operate." Um, and so I think that, you know, is being forced to be shifted away in this new era where, where people can produce so quickly.

    7. AG

      And then you made this provocative claim around ratios, so just let me understand this. What i- do you think is gonna be, like... What is, I guess, the right ratio? 'Cause we can assume maybe that the future will adjust towards what is the right ratio in the future.

    8. GG

      Yeah. I, I think now a, a team of three or four engineers can produce the output of, of what a previous team of 20 or 25, uh, engineers could. And so I think, you know, a really small pod of, let's say, three to five at max engineers and a PM can be a good ratio, uh, because they're able to iterate and produce at a, at a much, much higher velocity, and you gain all of the benefits of being a smaller team, being very closely aligned, um, without the cost of just, like, you lose a lot of horsepower in, in the previous

  12. 45:45 – 52:03

    AI for your team vs AI for your customers

    1. GG

      world.

    2. AG

      So we touched on this a little bit in the podcast, but I'd love to get your take as somebody who's day in, day out building an AI company. What is the difference between AI that is ready for an internal audience and AI that is ready to scalably engage with customers, and how do you build that?

    3. GG

      Yeah. I, I think it's, um, about the, uh, you know, technology that you build around quality specifically, right? I think when you're-- Whether it's an internal audience or even it's a tool that's meant to generate outputs that you then review, right? The, the quality bar needs to be so much lower because you can feel confident that, "Hey, a human is gonna look at this before this gets, um, you know, shared in any sort of public way." But when you strip that away, right, you need to have the appropriate guardrails in place, and you need to be, um, you know, thinking about every update you make and measuring them against both those guardrails as well as, you know, continuing to do some level of human QA, right? So, you know, something as simple as, you know, bumping a version of a model. When you're building something that's internal or, or not directly going to customers, you can just do that and feel pretty confident that that's gonna work. When you're shipping something actively to customers, you need to have a much rigorous process, much more rigorous process around having an environment where you have, um, you know, sample data that you trust, being able to run, um, you know, uh, o- outside of production these generations on that sample data, ensuring that they're still, you know, passing guardrails at the similar rate that they were previously, doing some additional QA to make sure that some regression didn't happen that you weren't expecting. Um, and then, and only then can you feel confident, you know, upgrading that model. And so when you're doing that, you know, hundreds of times a week because you're updating and improving and, and optimizing these, um, customer outputs, um, across, uh, you know, a complex customer base and, and trying to optimize for things like, you know, conversion, um, you need to be able to do those sort of, like, sandbox environments tests at, like, super high scale and with, like, a great degree of confidence, um, that's just not necessary when you're not directly talking with customers.

    4. AG

      Hmm. What's interesting there is you use, like, kind of like traditional PM language around it, guardrails, testing environments, sandbox environments. You didn't, like, put a new label on the skill, like, "Oh, you need to learn ... this type of online eval or this type of method gradation, h- what would you say are sort of those new skills that you've had to learn in the, from the pre-AI building world to now building AI products 24/7?

    5. GG

      Um, I mean, I, I think the, um, I, I don't think the fundamentals actually change that much. I think, you know, obviously what changes is when you're dealing with a, a probabilistic tool, right, the, uh, type of testing that you're doing in those sandbox environments and the type of guardrails and evals that you're running change pretty drastically. You change from, hey, we have, you know, tests that our software team writes that ensure, you know, even the most advanced tests, right, ensure that when you're interacting with the product, things happen as you would expect. Now you're in a world where you need to be testing, you know, the nth possible, uh, iteration of how someone might interact with this-

    6. AG

      [laughs]

    7. GG

      ... and feel confident about what's gonna come out the other end. And so, like, uh, it, it ends up being a combination of, you know, those traditional tests that you would run, like, like deterministic tests that say, "Hey, is this output what I expected, um, from a, a discreetly measurable sense?" And then, you know, these more advanced LM as a judge evals that are doing, uh, you know, a greater degree of reasoning behind, um, you know, a much more complex problem like groundedness, or a much more complex problem like ensuring that the AI doesn't do anything that it wasn't s- explicitly given permission to do, right? So those are cases where it's like, hey, like, the AI really wants to please. And so-

    8. AG

      [laughs]

    9. GG

      ... you know, c- when customers ask for things, and, like, there are stories about this, um, from the early days of, like, I forget if it was Toyota or someone, like, got a free, someone got a free truck or the AI to say that they could get a free truck on their chat. Um-

    10. AG

      Yeah

    11. GG

      ... and so, like, those are much more challenging things because the AI and the LM as a judge want to, uh, please. They want to do what they're being asked.

    12. AG

      [laughs]

    13. GG

      And so you need to be thinking about ways to, you know, uh, tie those things back to something much more concrete of saying, "Hey, like, I want you to actually tease apart every specific action that the AI has taken in this response, and I want you to actually link back to the specific, you know, instruction that gave it, uh, permission to do that," right? So you have to get, like, it's not just, "Tell me if, uh, a- uh, the AI has done anything it's not supposed to," right? You have to go a, a, a significant layer deeper than that if you want to, you know, do this at a significant scale.

    14. AG

      So examining the specific traces where things go wrong is really important.

    15. GG

      Yeah. Yeah, I think that's, um, particularly important. Um, and I think also, you know, examining the specific traces where thing go, things go right, right? Like, the, the, I think you have the opportunity where the AI is being creative and doing things in interesting ways, and let's, like, the example I gave earlier, like, uh, let's say in a t- in a topic where almost no one ever converts after they bring this up, the AI response one time helps them convert. What can you learn from the approach that the AI took to get to that point, uh, is, is an equally big opportunity as, like, "Hey, the AI made a huge mistake here." Like, what can we learn from how

  13. 52:03 – 58:43

    Inside an AI-native startup, from whiteboard to PRD

    1. GG

      you can tie that back to how it specifically made that mistake?

    2. AG

      So in our pre-call, we were talking about how you guys work, and I think this is gonna be really illustrative and insightful for others because you have a unique place, right? You're not at Uber with 200 other PMs. You're a small startup. You guys get to operate and define your processes, uh, to the level that the current tooling allows. So what are the interesting things we can learn about what an AI-native startup looks like in the inside?

    3. GG

      For sure. I, I think a lot of what we've discovered is that, you know, it's really easy to build. Um, it's really easy to build quickly. It's also, like, pretty easy to build with, with pretty good quality. I think those are things if you have the proper, you know, engineering workflows, leveraging, you know, the, the latest AI tooling, you can get really good at. What's hard is, is still building the right thing, and so that's where you spend a lot more of your time. And we found it to be very v- really valuable to spend a lot of that time, um, you know, in discussion, right? So there's, you know, f- four of us on the team. Um, let's all sit down and talk through this problem space. Let's have someone, you know, obviously bring to the table an artifact that, um, lays out the problem, lays out the opportunity, and then let's work through, you know, what are the biggest challenges we need to solve? How are we going to solve them? What's the experience we wanna create? Um, you know, whiteboard that as well, and then you end up with an artifact, which is, you know, a detailed transcript that you've recorded from this discussion and a diagram either on, you know, a live whiteboard or, you know, a FigJam or something similar. Um, and you can feed that into, um, an AI skill that then goes and takes that, defines a much more detailed PRD off of that, creates the appropriate Linear tasks to, you know, project manage that all out into the, uh, into the appropriate work streams, um, and, uh, actually s- as the agents start churning on that where that makes sense, um, and kind of follows on with areas where we need to, you know, do more definition or have more discussion. And so you end up with, you know, a lot less time, obviously, like, directly developing or, like, sitting down at a keyboard and writing out a PRD and a lot more time actually having those real substantive debates about How the product should work, how it should evolve, what comes next, and things like that.

    4. AG

      So am I hearing this right? Instead of spending your time writing PRDs or writing front-end PRs as an engineer, engineers and PMs are spending more of their time in meetings sketching out, "This is what it should look like," making sure that the transcript has that information and they have pictures with that information, then the AI comes back with, "Okay, here's a working prototype," and you improve it and ship it from there?

    5. GG

      That's right. I, I think, and that's why it is, it has been very important for us, and I think in the future it's very important to have everyone at that table be capable, uh, fro- to a, to at least a certain degree from a product and a business standpoint, right? We want people who understand what we're trying to accomplish as a business. We want people who understand and have some level of product intuition. And so it's when you have those, like, disparate expertises, but at least the same baseline of understanding where those discussions can be really powerful.

    6. AG

      Hmm. So if you're at a s- larger company like Uber or Bolt and you can't, like, adopt all of that, what can you adopt? Like, what, are there specific tools within how you're engineering your harness, specific processes with how you're working with your product team to just operate a little bit more like this nimble AI-native new way?

    7. GG

      Yeah, I mean, I, I think you can adopt a lot of it because even if the, the expected output is, "Hey, we have this, this standardized PRD that we need to go then, you know, get approval and review from different parts of the organization," you can still model how you get to that PRD in this more, um, AI-forward, AI-native way, right? So you can still have that opportunity to work with, maybe it's not often your full team if it's a larger team, but maybe it's a s- a subset of people that you feel are the most important in making those types of definitions and decisions, um, have those conversations, and then have the, you know, the automation and the, the skills developed to, you know, automatically turn that into the format that you need for your, you know, corporate processes that you need to run through. So I, I think a lot of it is, is very usable. You just need to adapt it to, you know, what output, what outputs are expected from you as a team, and also the personnel that you have around you and, you know, what types of people are gonna be most effective in, in those more open-ended, you know, uh, product-oriented conversations.

    8. AG

      All right, guys. Greg Greiner's the name. LinkedIn is where you can find him on social media. Envoy is his startup where if you want to increase conversion rates or get closer to your customers and not vibe code this whole thing out, but choose a solution that has built it for you, you can check it out. Thanks for being on the podcast, Greg.

    9. GG

      Of course. Thanks for having me.

    10. AG

      All right, guys. If you want more content like this, do comment below who you want to see on the show next, what topics you want to cover next. Should we have more on AI research and discovery, or do you want execution and coding? Just let me know, and I'll see you in the next episode. I hope you learned as much from today's episode as I did. If you can do one thing that's totally free that would help the show, it would be to check that you're following on Apple and Spotify Podcasts, check that you've left ratings and reviews on those platforms, check that you're subscribed on YouTube, leave a like and a comment on this video, and then share it with your friends. We're trying to make better and better podcasts. After two years, we think we've gotten something pretty good going. So let us know what we can do to make it even better, who else we should interview, and we will put on the best shows we possibly can. Finally, don't forget my offer for the bundle. You get an entire year of my paid newsletter, plus my favorite AI tools, Bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arise, and Mobbin. That's $27,000 worth of value for just $150. So check that out at bundle.aakashg.com if it interests you, and I can't wait to share our next episode soon.

Episode duration: 58:53

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode JpyhPePe150

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.