Career frameworks, A/B testing, onboarding tips, selling to engineers | Laura Schaffer (Amplitude)

1. 0:00 – 4:15

   Laura’s background

   1. LSLaura Schaffer0:00

      ... like the, you know, dead of the night, and by that I mean, like, you know, 7:00 p.m. or something on, I think it was a ... Pretty sure it was a Friday. We just asked for forgiveness and kind of sh- put these questions into the sign-up form and ran as an A/B test with a small group. And, you know, I'm fully expecting, okay, this is gonna like hurt our numbers, but maybe it won't be so bad, you know. And I'm gonna be prepared to advocate the power of this data that we're getting. And I was totally ... and I'm thinking of it like written story to write like a, like the framework for how I wanted to, to surface this. And we start to get the data for this thing. I'm not getting an improved conversion. Like we ... There's no personalization, nothing past it, just the questions. It improved conversion by like 5%. Like, just improved sign-ups. And it's one of those, like, what? Like, okay, this, like ... What is going on here?

   2. LRLenny Rachitsky0:49

      (Instrumental music) Welcome to Lenny's podcast where I interview world-class product leaders and growth experts to learn from their hard won experiences building and growing today's most successful products. Today my guest is Laura Schaffner. The week we recorded this chat turned out to be Laura's first week in a new gig as head of growth for Amplitude, taking over for a previous legendary guest, Elena Verna. Prior to Amplitude, Laura was VP of product and growth at a company called Rapid. Before that, she spent over seven years at Twilio as head of growth and PM lead of the growth platform and the experimentation platform at Twilio. In our conversation, we dig into Laura's career growth framework and the importance of carving your own path versus waiting for one to be carved for you. We also get into a bunch of tactical and surprising advice around running experiments, making decisions on gut versus data, developing your growth strategy. And how to sell your product to developers. Laura has a wealth of wisdom and I learned a lot from our conversation. With that, I bring you Laura Schaffner after a short word from our wonderful sponsors. This episode is brought to you by public.com who want to tell you about their new treasury accounts which earn a 4.8% yield on your cash. That is higher than a high-yield savings account while still being backed by the full faith and credit of the U.S. government. Treasury yields are at a 15-year high, but buying U.S. treasuries is super complicated. You have to go to a bank or navigate an ancient government website, or at least that was the case. Now you can move your cash into U.S. treasuries with the flexibility of a bank account. You can access your cash whenever you want, even before your treasury bills hit maturity. There are no hold periods, no settlement days, just a safe place to park your cash and earn a reliable yield. Public will automatically reinvest your treasury bills at maturity, so you don't have to do anything to continue growing your yield. And you can manage your treasuries alongside stocks, ETFs, crypto and any alternative assets. Do all your investing in one place and earn 4.8%, a higher yield than a high-yield savings account. Only with a treasury account at public.com/lenny. This episode is brought to you by Eppo. Eppo is a next generation AB testing platform built by Airbnb alums for modern growth teams. Companies like Netlify, Contentful and Cameo rely on Eppo to power their experiments. Wherever you work, running experiments is increasingly essential, but there are no commercial tools that integrate with a modern grow team stack. This leads to wasted time building internal tools or trying to run your experiments through a clunky marketing tool. When I was at Airbnb, one of the things that I loved about our experimentation platform was being able to easily slice results by device, by country and by user stage. Eppo does all that and more, delivering results quickly, avoiding annoying prolonged analytic cycles, and helping you easily get to the root cause of any issue you discover. Eppo lets you go beyond basic click-through metrics and instead use your North Star metrics like activation, retention, subscriptions and payments. And Eppo supports tests on the front end, the back end, e-mail marketing and even machine learning clients. Check out Eppo at GetEppo.com. GetE-P-P-O.com. And 10X your experiment velocity.

2. 4:15 – 5:50

   How to carve your own career path, and an example from Bandwidth

   1. LRLenny Rachitsky4:15

      Laura, welcome to the podcast.

   2. LSLaura Schaffer4:18

      Awesome. Thanks, Lenny. It is so great to be here. Thanks for having me.

   3. LRLenny Rachitsky4:21

      It's great to have you. So I asked Elena, Elena, Elena ... I'm not even sure how to pronounce her name. Maybe you know.

   4. LSLaura Schaffer4:26

      Elena, you got it.

   5. LRLenny Rachitsky4:26

      What is it? Elena, okay, okay. I always ... I think I've said it wrong all the time, all this time. Okay, Elena. So I asked Elena Verna, who's a popular guest on this podcast, who I should have on this podcast, and you are the first person that immediately came to mind.

   6. LSLaura Schaffer4:39

      Mm.

   7. LRLenny Rachitsky4:39

      And so I'm really excited that we're doing this and that you agreed to be on.

   8. LSLaura Schaffer4:42

      Well, she's the best and I'm really happy that she referred me, 'cause I'm just stoked to be here. So thanks for listening to her guidance.

   9. LRLenny Rachitsky4:50

      Absolutely. And it's kind of a cool time to be chatting. You're kind of the, the newly-minted head of growth at Amplitude, and so congrats, first of all.

   10. LSLaura Schaffer4:57

       Thank you. I appreciate that. Yeah. Well, this is my, um, day two and a half here.

   11. LRLenny Rachitsky5:03

       Wow.

   12. LSLaura Schaffer5:03

       So very new.

   13. LRLenny Rachitsky5:03

       You're a veteran.

   14. LSLaura Schaffer5:04

       That's it. (laughs) Yeah, right.

   15. LRLenny Rachitsky5:06

       I love that some companies there's, like, a little percentage that shows you how many people have joined before you and I wonder what that percentage already is at Amplitude.

   16. LSLaura Schaffer5:12

       Mm-hmm. We had that at Twilio and I got pretty, pretty high up there. After a while we had, like, a stack rank and a spreadsheet. And yeah. But it's, it is funny. So I ... Wherever, (laughs) wherever that thing is that's in Amplitude, I am, like, right fresh there at the very bottom. So...

   17. LRLenny Rachitsky5:25

       (laughs) What was the number you got to at Twilio? Any ... Do you remember?

   18. LSLaura Schaffer5:27

       Um, yeah. No, I was, I was very proud to crack, like, the top 50. That was, like, my claim to fame. 'Cause as people, like, left, right, you kind of, like, move up, right?

   19. LRLenny Rachitsky5:36

       Right.

   20. LSLaura Schaffer5:37

       Um ...

   21. LRLenny Rachitsky5:37

       Yeah, bittersweet.

   22. LSLaura Schaffer5:39

       So ... It, it ... Well, yeah, right? It's kind of a ... It's like on one hand it's like, oh, very cool, I'm like one of the OGs. On the other hand, it's like, oh my gosh, like, this person's gone up? Like, that's a bummer. Just a shift, but, um, I'm excited about it, for

3. 5:50 – 10:18

   Laura’s career growth framework

   1. LSLaura Schaffer5:50

      sure.

   2. LRLenny Rachitsky5:51

      So you have this new exciting role and I thought it'd be fun to start to chat about career growth and just how you think about career growth. I know you have a framework of how you think about your own career growth, and clearly it's worked out.So I'm curious to hear about it and see... And see how it could be helpful to folks that are, that are listening. So yeah, can you just tell us about how you think about career growth?

   3. LSLaura Schaffer6:11

      Career growth is often, right, like it's, it's definitely not a straight line up, um, but there's definitely some kind of frameworks and methods that have worked really well for me. And I think to dive into it, it's first good to just talk about the one that I most typically see people use to try to grow their career, and, and why that can be a little problematic, which is that I see most people are, you know, try to work really hard at kinda the job that they have, right, within the role that they're, that they have at a company. Do whatever you can to grow there, show your manager all these things. I've seen people who keep spreadsheets of sort of wins so they can come up with performance reviews. You know, maybe try to get better advocating for yourself, maybe try to get peers to notice, or your managers' peers. And that's all good, like that's all good stuff, but the problem with it is that you're limited to what your manager's ability is to advocate for you, to promote you, and you're also limited by the explicit trajectory of your role at that company and, and where the, there's room for that around the company. And then often that perception can sometimes be a little bit in contrast to what your perception is, right? And also, other things that happen, like your, you know, your manager leaves, and then you kinda have to restart with someone else, right? So the method that I use kind of tries to take that power back a little bit, and something that I learned really early on in my career, and was very lucky to learn by accident, was at a company called Bandwidth, which is, like, my first kind of, quote-unquote, real job. And, you know, Bandwidth is now a public company and they've done all kinds of crazy amazing things, but I joined when it was just 50 people, and I actually joined in sales. And I was, you know, just hungry to kind of make it succeed and grow, and, you know, bright eyes and everything, you know, first kind of real job. But I realized after a few months of being in sales that I was often, like, repeating the same thing over again, like, using the same thing to sell over and over again. And it's like, gosh, like, this isn't ideal for the customer 'cause they're gonna call me and ask me these questions and wait and get these answers and all this stuff. And it's not ideal for the company 'cause they're, like, paying commission on this every time. Like, that's not gonna be efficient for our growth. And because we were small, I was able to catch our GM, and I was just like, "Hey, um, you know, I've noticed this pattern where, like, I'm repeating things, kind of, over and over again, and, like, they're asking the same thing. Like, I think we should put that online. I think we should make that available so they can just, like, see it and then buy it," 'cause we had an online checkout process. And I was kind of expecting him to be like, "Oh, well, I know that's important, but, you know, for this to be worth another, like, we need to, like, you know, we need to do it this way," and obviously he thought all about it. Kind of thinking like, oh, I'm gonna come in, like, this new person, he was gonna help me understand what I'm, like, missing here. There's, like, a little bit of that, that I was expecting. And he goes, "Wait a minute. Like, tell me more about that. What do you mean?" And by the end of the conversation, he was like, "Hey, why don't you, like, go do that? Why don't you go build that, like, experience? Why don't you put that stuff in, like, in, on a, on a self-serve flow?" And we called it e-commerce manager, and it was, like, kinda growth before there was growth. This is, like, 2010. And that, that moved me into a totally near, uh, new position, and the main learning that I had from that was, which really kind of took life at, at Twilio, and I, uh, absolutely worked for me there, I'm happy to talk about that too. But the, the core of that learning was your executive team, and executive teams at companies are often very sharp. But the nature of their day-to-day just does not link them with customers, right? And that means that over time, especially as a company grows, they often lose access to some of the best insights, and you know, the heartbeat of the people they're providing value to, in contrast to folks that are closer to the problem. And so that means that your superpower is in really pulling those insights in and bringing them to life, staying close to the customer. There's not a single leader or executive that isn't gonna be stoked to hear about valuable customer insights that highlight problems they might not be seeing. And there's a lot of those. So, especially, you know, when they align to North Star metrics, those, those ones are, are sort of the powerful ones. That was the way that I grew my career at Twilio, and I'm happy to share, kind of, that journey too.

4. 10:18 – 12:25

   The value of customer insights

   1. LSLaura Schaffer10:18

   2. LRLenny Rachitsky10:18

      Yeah, it'd actually be cool to hear maybe another example of that. But I think-

   3. LSLaura Schaffer10:21

      Yeah.

   4. LRLenny Rachitsky10:21

      ... I think an interesting thing that comes up for me here is, sometimes you may have an awesome idea, and it may not immediately happen. It may not be like, "Yes-"

   5. LSLaura Schaffer10:29

      Mm-hmm.

   6. LRLenny Rachitsky10:29

      "... Laura, let's move on this right immediately." And I think it's important to just recognize, like, they're not gonna follow all your ideas. Uh-

   7. LSLaura Schaffer10:36

      Right.

   8. LRLenny Rachitsky10:36

      ... but they're always looking for better ideas. And they, like, to your point, they may not have the information that will lead to an idea that you will have 'cause you're, like, on the ground dealing with real problems day-to-day. So I think it's important to recognize you're not gonna always get your way, and that's normal.

   9. LSLaura Schaffer10:52

      Yeah, totally, and right, and it's- it's- it's kind of about, like, almost, like, building up your, your individual, like, brand a little bit. And I think one of the most powerful and accessible ways to do that is learning about your customers. There's always those people at, at companies where it's like, "Oh, well, you know, I mean, she just knows our customers," or, "He just knows our customers. They just know our customers. Like, they just know." And it's like, well, how do they? They just know. Like, let's ask that person. Let's get their feedback. And those people often have a good amount of, kind of, you know, brand recognition and power within the company, and they're often thought of when the company needs to do something new or different or, you know, if someone is, you know, hiring, maybe they're thinking about that person for, like, a cross-team thing. So it's- it's one of the ways that you can kind of build, build that, that brand. And again, it's, it's, I think it's a sweet spot because it's something that is very valuable to all, everyone, all the way up to the most senior leaders, which we can talk about here in, in a minute. And so it's going to be valuable for you and a valuable tool, kind of, no matter where you're at in your career. And yeah, and that's not always an immediate payoff, but it often does give you a trajectory outside of just your role and just your manager. It gives you something a little bit broader.

   10. LRLenny Rachitsky12:05

       So maybe a simple way of describing... To, to kind of mirror back what you're saying is kind of carve your own path. Don't necessarily assume your managers will give you the path that makes most sense for you, or even give you the biggest opportunity. Just, like, propose, "Hey, I think this might be a better opportunity and I'd love to pursue it." I'd love to hear the Twilio example, if that's generally,

5. 12:25 – 16:14

   The “voice of the customer” report

   1. LRLenny Rachitsky12:25

      um-

   2. LSLaura Schaffer12:25

      Yeah. So I actually joined... When I joined Twilio, there was no growth team at all. Like not even a breath of it. Uh, I joined in product marketing and I was leading our, uh, product marketing for our messaging lines. And, you know, but I followed the same guide that I just mentioned. I made it a p- you know, my own kind of personal policy to like, "Hey, I'm gonna do my job and I'm gonna do well and keep notes of things I'm doing well," and all that kind of stuff, 'cause it's good. But I'm also gonna get to know our customers and I'm gonna get to know our customers really well. And I'm gonna pay attention when I'm connecting with them, not just about the space I'm in, but just broadly what are some of the pain points and things they're articulating that, that are relevant to the business and we're trying to get done. And one of the things that came up was that users were struggling and folks were struggling to get started and use Twilio. And that contrasted so deeply to some of the things that our executive team was saying, and, and had high conviction and our company had high conviction, which is that Twilio was so easy to use. In fact, it was like top three things about Twilio that we were really trying to get out there, man. We're so easy. Developers love us. They say we're so easy. And there's always, like... There were tweets coming all the time, like, you know, developers saying like, "Oh my gosh, like, I got started in a couple minutes." So there's all these things that kind of made that compound and made that conviction stick. But as I was talking to customers, I was hearing a very different story and it made sense. As we were penetrating new markets, adding more products, we were adding complexity and we were pulling in folks who are a little bit less motivated, and those things contributed to people saying, "This is... This is difficult." And so, you know, at the time, you know, this, this wasn't a, you know, 50% of my record. I could just kind of go to the floor and like go to someone and be like, "Hey, like, there's this thing I heard about, like, I think we should do something about it." But there was another tactic that I could take, and this is... I just started sharing a voice of the customer report. I started sharing my insights, started writing them down and just sharing them, and it became this kind of digest. And eventually people were like, "Hey, can I... Can you share this with me? Can you share this with me? Can it be on your list? Can you share it with me?" And this is in like a few months of me joining that I was doing this. And then that turned into, "Hey, you should, like, host a quarterly voice of the customer session for, for all of product." Right? And, and, uh, and this was a request that was coming from some of the senior leaders at the company. And when our... Uh, Jeff, Jeff Lawson is our CEO at the time, heard about it, he started attending too. So now in the session I started pulling in other people's insights too, right? 'Cause this was, these were... You know, now they had a forum for this, I could kind of do that. And, um, and, and, and, you know, have people send that to me and I could compile it and all of these things. And so then, you know, this kind of established me as that person who knows about the customer, even after short tenure. And then, you know, when, you know, came time to do annual planning, that, that year, and I joined in 2014 at the end, so this is 2015, I pitched this idea. "Hey, we think that it's easy. It is not. Here's data that I have, the information that I have, and I think that we need to start a growth team here and that needs to be a core focus." And I was able to bring in really critical, um, you know, kind of partner to that and other folks who could support that, because I'd built up some of that trust. So by the time I was making that pitch, I had someone like, you know, En- uh, Andre Crow, who was like the seventh hire at Twilio and got to, like, number three on that spreadsheet or whatever, who was really close to our CEO being like, "Yeah, like, we desperately need this." I'm seeing this... He, he led a, you know, website. He basically created the Twilio brand and he, like, led all the website stuff and he's like, "Yeah, we definitely need this." So not only did I have that kind of little bit of trust, right, from the executive team, but I also had folks who were just trusted on their own advocating and supporting this that I was doing. And so it was approved with, like, almost very easily. I mean, I put stuff together for it, but it was, um, you know, kind of the meeting before the meeting had already been done by those other things. So, it helped me create the growth engineering, growth product team at Twilio.

6. 16:14 – 18:16

   Leaning into your strengths

   1. LSLaura Schaffer16:14

   2. LRLenny Rachitsky16:14

      I love just how proactive your advice is here. There's a lot of people that don't do well and then just like, "Oh, I never had the opportunity," or kind of get kept... Got looked over all this time. And I love that there's all this just like, "Here's things you can be doing to get in front of people to provide value, to just create opportunity for yourself." Any other advice along the lines of just like, "Here's the kind of things you could do for yourself versus waiting for someone to come and give you opportunity"?

   3. LSLaura Schaffer16:42

      Yeah, I mean, I think that's the most easily actionable, because to do all of our jobs, we need to know customers. Like, we need to know about customer insights product, we need to know. But then also customer facing teams, so for those who want to crack into product. You and your insights are extremely valuable. You're talking to customers every day. You know more about their problems and their pain than a lot of other people do. And so, you know, that's... That is by far and away to me the, the most powerful and accessible one by e- by anyone in any role in any space. But, you know, I'll also say that, you know, that broader concept of just, hey, like, there's things that you know and things of value that you know that others can benefit from at your company. And, and building your brand as someone that is supportive, smart, creative, able to solve problems. You know, make sure that you're sharing that, right? And so, you know, maybe you're really freaking good at communicating with brevity. I suck at that, by the way. So like, more power to anyone that can do that. I'm actively working on it. So, so share that. Like, go, go to your general Slack channel or whatever and just be like, "Hey," like, just run up, like, some tips for how to do it. You know, "Some ways that I'm good at this." And those kinds of things can really go a long way towards people starting to view you as, you know, an SME and not just the space that you're in, but in sort of broader areas. And that can always present and open doors for you when other people are sort of looking up to you and seeing you as someone who's, who's strong in ways outside of just the role that you're in.

   4. LRLenny Rachitsky18:07

      And SME is, uh, subject matter expert. Is that right?

   5. LSLaura Schaffer18:10

      Mm-hmm. Yes. Gonna-... tha- thank you for unpacking my acronyms. That's another thing that I am actively working on.

7. 18:16 – 20:20

   The experiment that shifted the way Laura thinks about friction

   1. LSLaura Schaffer18:16

      It is not-

   2. LRLenny Rachitsky18:16

      I, I got you.

   3. LSLaura Schaffer18:17

      ... hard.

   4. LRLenny Rachitsky18:17

      I'll be on the lookout.

   5. LSLaura Schaffer18:18

      Thanks.

   6. LRLenny Rachitsky18:19

      Um, maybe one last question along these lines is-

   7. LSLaura Schaffer18:21

      Yeah.

   8. LRLenny Rachitsky18:21

      ... do you have any advice for framing the proposal, framing an opportunity to, you know, your manager or higher-ups that you see has worked best broadly?

   9. LSLaura Schaffer18:31

      Yeah. So, yeah, and, and one thing I wanna say too is like with this stuff, I don't think that it necessarily does go, like, counter to what your manager is doing. It's more like supporting them, like, right? Like, I've done this kinda stuff and then it's helped my manager promote me, right? So it's, it's not necessarily, "Oh, well, do this if your manager is, like, failing you," or they, you know, are not that bored in you or they can't support you. It's more like do this because this is gonna be an accelerator for yourself irrespective of your manager, but then also it'll be an accelerator for your manager in supporting you. 'Cause one of the things that, you know, that comes into play a lot when managers are figuring out promotions and doing all those things is, like, they'll sit in a room off on calibrations and, you know, with a bunch of people, and it makes it a lot easier when those people have had some kind of a- access, or exposure, or whatever to you, um, in a positive light. So, these things can all run, you know, with, you know, your manager and, and, and not against, but it's just another way of you kind of taking the, taking back the ability to build that momentum instead of relying on all of that going through one single other person.

   10. LRLenny Rachitsky19:34

       What I like about your second example is you just did it. You just started doing that kinda report for the company. It wasn't like, "Hey, I have a proposal. Here's what I think you should do. Should we do it?"

   11. LSLaura Schaffer19:42

       Exactly.

   12. LRLenny Rachitsky19:42

       It's just like, "Yeah, just do it."

   13. LSLaura Schaffer19:44

       Yeah. Like, uh, ungate your knowledge, I think is the, the buzzword that I'm, I'm hearing, right? Um, and you could do that-

   14. LRLenny Rachitsky19:50

       Mm-hmm. I've heard of it.

   15. LSLaura Schaffer19:50

       ... within your, uh, I think that's an Elena-ism. Like-

   16. LRLenny Rachitsky19:54

       Mm-hmm.

   17. LSLaura Schaffer19:54

       ... we'll see if we can, how many times we can, we can bring her up. But right, it's just, you know, like, you can do that within your own, you know, company. Like, everybody, everybody is skilled at things that they aren't, you know, explicit to their role or their space. And I think that ungating that opens opportunities. And if you're not sure, then go to my favorite go-to, which is talk to customers, get insights. Those are incredibly valuable, so rarely do people share those when they find them. So, so be the person that, that does

8. 20:20 – 28:53

   Questions that improved Twilio’s onboarding and conversion rate

   1. LSLaura Schaffer20:20

      that.

   2. LRLenny Rachitsky20:20

      Another area I wanna chat about is experimentation, and growth, and data, which, uh, makes sense you have strong perspectives on being the new head of growth at Amplitude. So, maybe we start with experimentation.

   3. LSLaura Schaffer20:35

      Mm-hmm.

   4. LRLenny Rachitsky20:36

      You mentioned that you had, there was, like, a really interesting surprising result in an experiment you ran at Twilio that kind of maybe changed your perspective on experimentation and what you think might work and not work.

   5. LSLaura Schaffer20:46

      100%, yeah. I'm sort of fortunate to have kind of two mind-blowing experiments that really shifted-

   6. LRLenny Rachitsky20:51

      Mm-hmm.

   7. LSLaura Schaffer20:51

      ... the way that I think about growth. So one of 'em, one of my favorite ones, happened very early on at Twilio. So after I'd c- created this, this growth team, like, one of the things that I saw as, as to me an issue, was that at our sign-up flow we, we just asked people for a username and an email and, like, a password, and that was it. And that's actually relatively common at the time, and this is kind of a while ago now. Like, everybody kind of started doing segmenting users. But we didn't, and we actually, there was a lot of existing conviction around that. It was like, "Okay, we, we're targeting developers. Developers, they an- you know, they just wanna do, they just wanna get their hands on things. Don't put anything in their way, it's gonna be disastrous. We don't want any shenanigans here with these, these folks. Let's just let them in the gates." But to me, this was a really big assumption to make, and, and a very costly one. It's like, "Okay, if, if that's the case, we don't know, we're not gonna know anything about anyone." And we didn't know who was signing up. We didn't know what they wanted to do. And that hurt our ability to understand how people were performing from a quantitative perspective. You know, we were a little bit lost with prioritization and there was a number of implications here. But it was obviously a very contentious kinda space. So, I, you know, this was the very first thing that I, that I did on the first experiment that I ran. I, um, I did, you know, some research and understanding, okay, what are the most important questions to ask? Like, what would I do, really, really need to know? And it was stuff like, you know, what, what language are, you know, are you coding in and what's your use case? What product do you wanna use? And then, you know, there's, you know, one around, like, are, are you a developer at all or are you something else? 'Cause there was sort of rumors that we were having not just developers sign up, which is this whole other interesting kind of story. And I, you know, thinking these questions that would also, you know, potentially be things that, that our developers signing up would sort of understand why we're asking, that it would feel natural. And, but anyway, you know, again, like adding anything to the sign-up was very contentious. But I really just wanted to get a little bit of data on it. So, I wanted to run a test. I didn't have a team. We didn't have an engineering team yet and none of that stuff had been built out. It was just me, myself, and I. But like I said, I had started to, to build a little bit of trust and pulled in good old, um, Andre, who I mentioned earlier, who because he was early employee and just he had, kinda had access to everything, like one of those, those people. And he had also, was, you know, supportive of, of this and kind of had similar hunches. And so, in like the, you know, dead of the night, and by that I mean like, you know, 7:00 PM or something, on I think it was a, pretty sure it was a Friday, we just asked for forgiveness and kind of sh- put these questions into the sign-up flow and ran it as an A/B test with a small group. And, you know, I'm fully expecting, okay, this is gonna, like, hurt our numbers, but maybe it won't be so bad, you know, and I'm gonna be prepared to advocate the power of this data that we're getting, and I, I was totally geared up thinking about, like, written, started to write like a, like the framework for how I wanted to, to surface this. And we start to get the data for this thing, and I'm not kidding, an improved conversion. Like, we didn't, there's no personalization, nothing past it, just the questions. It improved conversion by, like, 5%. Like, just improved sign-ups. And it was one of those, like, "What?" Like, "Okay, this, like, what is going on here?" And, I actually dug into it, and what I found from just talking to a few customers that went through the flow and just, like, learning about how they felt about it, it was actually, for folks, it was, like, comforting. You know, when you think about it, when, when users are signing up for your product for the very first time, like, it's new, right? This is new. That means it's scary, they're, they're expecting it to be difficult.... they're anticipating that it- there's gonna be friction and challenges, and, and that they're not gonna figure it out. They're almost like looking for the bogeyman, right? And that's the head space. It's often the head space that any of us are in when we're doing something new for the first time. Like, "Ooh, this is, could be very challenging." And so by putting in these questions, it's like, "What's your language?" It's like, "Oh, like I do, I, I, I code in JavaScript and, and I'm, I can select that. Well, that's something I'm comfortable and that would make my, my journey easier." Like, yeah, bingo. Oh, and you're, you know, that's, that's my use case. Okay, like I'm in the right place here. Like, you know? It was, it was actually giving folks something comforting and, and challenging the notion that this was going to be difficult, just the questions, because it was aligning to some of the things that they were organically thinking about, which is, "What if they don't support my language?" Or like, "Can I even do this use case I wanna do?" And so it was just a really interesting, you know ... The takeaway for me for this, like the really interesting takeaway was the psyche of the user is so, so critical, right? Like, that's just as important as understanding your product and the broader, like, market you're applying to and all those things. Like, just the psyche of, of users, new people doing things for the first time with your, in your user flow. Like, understanding that is powerful. You know, and the simple, like, kitschy thing I say is that, you know, ultimately the learning here was bad friction is bad and good friction is good, right? There's n- no such thing as si- being simple, uh, it's just all friction is bad, which is sort of kind of what I had assumed going into this.

   8. LRLenny Rachitsky25:50

      I love that you, uh, you were new to Twilio and you just YOLO'd an experiment to production.

   9. LSLaura Schaffer25:55

      YOLO'd. Yeah.

   10. LRLenny Rachitsky25:56

       That's, uh, that's a big move.

   11. LSLaura Schaffer25:58

       It ended up being, like, very, you know, very helpful for everyone. I, you know, shared the insights from it and, like, all these things-

   12. LRLenny Rachitsky26:04

       Yeah, yeah.

   13. LSLaura Schaffer26:04

       ... and shared the data

   14. LRLenny Rachitsky26:04

       And, and it drove conversion, huh?

   15. LSLaura Schaffer26:06

       But for sure, like, you know, used such, you know, processes with, with caution, uh, for sure. Yeah.

   16. LRLenny Rachitsky26:13

       I love it. That's amazing.

   17. LSLaura Schaffer26:13

       The, the right way to do this, we lo- like advocating for, for the engineers here is the right way to make any changes in production is, is, you know, through or with the approval of engineering. But it was, um, it was, it was the right move overall and definitely helped business, right? So, um, yeah.

   18. LRLenny Rachitsky26:28

       Yeah, no, I love it. No, that's great. I like that move. I think we need-

   19. LSLaura Schaffer26:30

       Yeah.

   20. LRLenny Rachitsky26:30

       ... more of that probably. I wanna dig into which actually ... So what is it you changed? You added how many questions, and then what were the questions?

   21. LSLaura Schaffer26:36

       There was a question around, like, what language are you coding in? And then as an option to that, I was like, "Oh, like, I'm not, I'm actually not coding, like I'm not a dev- developer." Like, so that was, uh, it, you know ... For us it actually gave us two really, really interesting data points. One was, like, you know, how many developers versus people who are not coding were in our flow, and then what language are they coding, which was massively helpful not just for, like, growth and onboarding, but like our documentation team, our docs team. Figuring out, like, what we should ... You know, that ended up being a critical way for us to gauge trends over time and catch things before, like, you know, whatever, you know, reports would come out at the end of the year of what people are doing and we could sort of see it. And then also product, what product are you interested in using? That was very critical for knowing kind of the basics of how to organize someone's onboarding, right?Are you doing SMS? Are you doing voice? Are you doing, you know, just two-way, one-way, whatever. And then use case. And use case is like, are you doing appointment reminders or are you doing, like, a, you know, autoresponder? Or are you doing anonymous communications, like for a dating app or something, right? So, those were, those were the very first questions.

   22. LRLenny Rachitsky27:39

       Wow. Okay, so it was like four, four dropdown questions and that increased conversion. I love these examples where friction add increased conversion. Like, there's so few of them. Like, you hear about this could work, and it's rare. And so what did you take away? Like, like, what's the pattern you took from this? Like, there's the idea it's good friction, but is there something that you're like, "Here's what is a sign if this is gonna be good friction"?

   23. LSLaura Schaffer28:02

       This still alleviated a problem. It alleviated the problem they had where they were coming in and worried that it was gonna be difficult or that they weren't gonna be able to figure it out, they weren't gonna be able to get their footing. And I'd say that that's not unique to Twilio. That's, that's something that I think users experience at any front door at any company, any sign up, beginning of the sign-up flow. It's like, here we go, right? Like, buck- like, buckle up. Especially when, you know, it's in a work context and, you know, there, there might be extra pressure on you to succeed or, you know, for you to make an as- a- an accurate assessment, right? So you're just kinda, you know ... I think that psyche of like, "Okay, am I, like, in the right place? Is this gonna do what I need it to do? Like, c- can, can I figure it out? Am I capable?" Like, that's ... These are extremely common things for people to feel when, when they're signing up. And so, you know, certainly, you know, that I think can, can carry it to any, any place.

9. 28:53 – 31:26

   Thinking about the psyche of your users

   1. LSLaura Schaffer28:53

      I, I'd encourage absolutely everybody to be putting those, those kind of experiences within their early onboarding, not just, you know, for you selfishly so you can learn and segment them appropriately, but also so the user can feel more confident as they're getting going. Like, "Hey, uh, I'm in the right place. This is gonna do what I need it to do." But I think that the, the carryover there is just the, the, the psyche of the user, right? And just being so aware that it's not so cookie cutter as what is the problem my market experience is, and what can my product do to help them? There's also this other thing in the room, which is so important to people's success and their ability to succeed with your products and your self-serve experiences, which is what is the mentality and the psyche of the person at the various stages in your journey? And if, you know, if you're, if you're not incorporating that or addressing that, you will absolutely miss things or things will fail and you'll be very confused as to why. We had a great, um, experiment that I'm happy to talk about where-

   2. LRLenny Rachitsky29:52

      Yeah.

   3. LSLaura Schaffer29:53

      ... like same, same concept but totally different situation, which is later, you know, i- in, in onboarding. One of the things that, that we tried to do over time to make Twilio less complex was to offer steps, like, you know, you see this a lot in onboarding. Like, s- welcome, step one, like here's what we wanna build. Great, we all know that. Now, okay, step one, go do this thing. Step two, go do this thing. Three, do this thing. Four, this thing. Five, bam, you're live. Congratulations, aha, all these things.So, you know, we shipped that, you know, got that out there and I was like, yeah, it was like, i- it improved conversion but it wasn't, like, that great. And it's like, man, like, we went from, like, there being, like, absolutely nothing, like, choose your own adventure, figure it out, go figure it out, g- good luck, to this kind of prescriptive thing and it, like, wasn't converting as well. And, you know, sort of talked to some users and there wasn't anything particularly obvious that was coming out as to, like, what the issue was. It was like, "Oh, yeah, like, I do step one..." And we did, like, mocks with eight people, like, "Okay, now I, I do step two." But there was one thing that, like, I was hearing that was coming out that was like, feels like something, and that was that the telephone number, the telecom part. Developers, when they were coming into Twilio, was like, things that were familiar to them, APIs, the language they're coding in, code samples, documentation, things that... like, the bogeyman, right? The things that would, like, psychologically trip them up. Telecom, phone numbers, things that, like, you know, the- these things that just were completely out of the zone of anything that they'd ever worked with before,

10. 31:26 – 33:58

    The hot dog analogy for burying “scary stuff”

    1. LSLaura Schaffer31:26

       especially, you know, early- earlier on in Twilio's journey, but even now, right? Telecom's a very different beast for most developers. And guess what was step one? Get a phone number. Because that's step one. Any time that anyone's trying to teach them how to use Twilio, like, one-on-one, they're always gonna sit down next to them and be like, "Okay, like, here, we're gonna go get a phone number and configure it," and that's what anyone every time will do. However, in a self-serve experience when you don't have that safe person sitting next to you being like, "Don't worry, it's gonna be okay. I'm gonna take you through this crazy telecom journey," they're on their own p- with that psyche, telling them, like, "Oh, my God, telecom? Oh, I can't do that." That sounds scary. Like, getting a phone number configured? Like, whoa, I'm out of my depth. Right? And so what did we do to test this out? Like, test out whether that was the issue? Actually, and it's first one in the MVP, like, kicked them out of the, the portal entirely and put them into a docs page where we could kind of manufacture an experience where the first thing they saw was code, and they're in the docs, safe place, the language that they're coding in. And then, like, snuck in there, it was like, "Oh, get a phone number. Like, let's go configure it," right? Not as, like, step one, not as, like, the leading thing, but sort of embedded. And the analogy I have for this is, like, pill in the hot dog. So, like, if anyone ever has, if you've a, if anyone's got a dog or-

    2. LRLenny Rachitsky32:39

       I do.

    3. LSLaura Schaffer32:39

       ... an animal that you have to feed a pill to, it's like, you can't just feed the pill to the animal, it's never gonna happen. But if you shove it inside of a hot dog, which, like, looks good and, mm, that's exciting, then you can get them to consume it more easily. And so this was-

    4. LRLenny Rachitsky32:53

       Yeah, we do that. We do peanut butter. That's our move.

    5. LSLaura Schaffer32:55

       That, yeah. Exactly, right? Yeah. Hot dog, peanut butter, all, all that, right? You, you wo- you, you kind of bury it, right? You embed the, like, scary, unpleasant thing. And so that's what we did with the phone number stuff-

    6. LRLenny Rachitsky33:06

       Mm.

    7. LSLaura Schaffer33:06

       ... that telecom stuff. And guess what? Even though we were, we couldn't mount out the, the, you know, console and they were going off and, and we had no easy return button, it converted better because we were addressing the big problem that was there at the time, which is their psyche. They were not ready to come in and immediately be thrown into a phone number experience that was, you know, letting the bogeyman out to party, and that's not what was gonna work. We needed to, you know, put that bogeyman pill in the hot dog, and so, you know, then, you know, once that validated, then we can actually go through the, the business of putting that into the, the onboarding flow correctly and then that converted even better. But, yeah. So again, the psyche of your users, such a critical thing to be thinking about. And if something, like, very logical isn't converting well, sometimes it means that you're battling against the psyche of a user and you wanna take a step back and, you know, think about and learn about where someone is psychologically in your

11. 33:58 – 36:21

    Why it’s better to be iterative and why experiments fail

    1. LSLaura Schaffer33:58

       space.

    2. LRLenny Rachitsky33:59

       Feels like you had this experiment that was, like, a complete redesign of the onboarding flow and that didn't work. And then your second attempt was, like, a different approach that's, like, a full onboarding flow. And I'm curious, do you have a take on just when you run experiments, and this is something we dealt with a lot at Airbnb and other places, like, do you just redesign the whole thing? Or is it better to iteratively work from where you are today and just experiment piece by piece towards some future, much better experience?

    3. LSLaura Schaffer34:26

       Here's what I'd say to this, is that from a high level, it's always gonna be better to be iterative. And the reason that it's better is that roughly 80% of the times, or some of the time, our hypotheses and the things that we believe will be true, aren't true. And this is, like, amazing. I mean, the, there's an amazing article out there, I'm happy to share with you so you can put in the, the, um, show notes-

    4. LRLenny Rachitsky34:48

       Yeah, absolutely.

    5. LSLaura Schaffer34:49

       ... that really takes a scientific approach to proving that out. Companies like Netflix and Microsoft, just over and over again, 80-plus percent, some companies say 90% of things fail. And so when you, the, the, the closer you get to something that is, you know, you kind of go bury your, your, you know, head in the sand or go into an attic and build something for six months and ship it, the more likely it is that you are gonna ship the 80% wrong stuff, right? Whereas the more iterative you are, the more likely it is you're gonna catch it sooner. And, you know, failure doesn't have to be a wall, it can be a compass, right? It can be the thing that leads you to the right thing. And so, you know, you always want to, as best you can, get stuff in front of customers so that you can get that compass and, and, you know, get, get that compass activated, know where to go. And, yeah. And so that means doing, you know, ugly things. I, I tell my teams all the time, "If it's not embarrassing, you've gone too far." Right? Got to be embarrassing. The first thing, "That was embarrassing. Kicking people out of, you know, onboarding..." Spent all this money and, like, whatever to get them into your sign-up flow and then the first thing you do, "Get out of here." Right? I mean, that's, that's nuts. But if it hadn't validated, that would've been a very cheap but very valuable learning. Instead, it was a very powerful, cheap learning in the other direction where, okay, now we know we can invest in it. We know that's the right thing to do. So always better to be iterative so that you are letting failure work for you instead of having it be a trap that you fall into.

12. 36:21 – 41:58

    Saving money by validating fast

    1. LRLenny Rachitsky36:21

       I know that stat you just shared is like per experiment, you're probably wrong 80% of the time. In my experience, launching a whole redesign is, is negative 100% of the time. I've been, I've grown weary to avoid that as much as possible, which is like, you kind of know that. You're taught that as you go into growth in product, but then you're just like, "No, come on. Let's make it awesome. Let's redesign this whole thing," especially, you know, your designer's always like, "No, let's start again."

    2. LSLaura Schaffer36:44

       Yeah.

    3. LRLenny Rachitsky36:44

       "Let's make it amazing." But it always ends up being negative, and you're like, "Oh, yeah. Well, it's too late now. We gotta launch this thing. We don't have time to-"

    4. LSLaura Schaffer36:51

       (laughs) Yeah.

    5. LRLenny Rachitsky36:51

       "... start again."

    6. LSLaura Schaffer36:52

       Well, and it's funny. I mean, in the articles then you'll see, like, it was, um, it was written by somebody from Microsoft who kind of built their presentation platform and did all these cool things. Like, as he went into actually trying to apply a scientific method of figuring out, like, how often people are wrong about their hypotheses and, and, you know, what they were planning to do, he's like, "I wonder if, like, that applies to us here at, like, Microsoft."

    7. LRLenny Rachitsky37:13

       Mm-hmm.

    8. LSLaura Schaffer37:13

       There's even, for him, that kind of question of like, hmm. And it, and it out, right? Like, it's just, you know, and I think it's challenging to, you know, when... 'Cause a lot of smart people in the world in this space doing things, and it's very difficult to think, "Gosh, am I really wrong, like, 80% plus, 90% of the time?" But when you think about it, makes total sense, right? 'Cause what has to happen for something to be successful? You have to understand the problem perfectly. You have to then understand who's having the problem perfectly. The customer. And what time they're having the problem. Then you've got to put the right solution in front of them to solve that problem. Maybe you've got the problem right and all that stuff right, but your solution is something off, right? Or maybe your solution is right, but maybe it's just not presented, i- not communicated in the right way. You could have any one of those things off, and it's not gonna succeed, right? It's not gonna have, you know, the, the metric impact you're expecting it to have. So, in that context, it's almost, like, incredible we do succeed 20 to 10% of the time given everything that has to line up. And so, I think it's one of those things where, you know, you really wanna go into it embracing that. "Okay, this isn't about how smart I am or how good my team is or any of that stuff. It's just, hey, the logic of this is a, you know, challenging to get it right, and let's embrace that, and let's, um, lean into, let's lean into, to that, that knowledge, and make it a part of our strategy instead of fighting against it."

    9. LRLenny Rachitsky38:38

       Have you found anything that helps you increase those odds? Or is it just this is the way of the world, and you probably can't significantly increase the chances your experiment works out?

    10. LSLaura Schaffer38:51

        So, here's the thing. I mean, I think, I think there's, there's very little that we can do to make that space easier, right? All those things have to be figured out. And so, I, I definitely think that you, uh, that, that everybody is gonna be in a space where their original, you know, ideas, untested ideas are gonna be around that hit rate. However, the way that you go about validating those can be totally different, and you can be very fast about validating those ide- ideas, and that's the key, right? And, you know, uh, A-A-B testing for, is one of the most expensive kinds of ways to validate an experiment, right? It, uh, you know, often requires design and engineering and the PM or growth person or marketing person who's crafting it, right? All these things are investments that take a lot of time, even for simple thing. And then you have the time factor. How long does the thing have to run to have, have an impact? So, all of that is extremely expensive. And so, I think the key is to just think through, "Okay, what are the things I can do to quickly validate what, what these ideas are that we and you can do that with painted doors, right? Which is where you test, right, the, the concept and the idea before it exists versus the actual experience. You can do mocks. Like, create, have, you know, create, if you've got a designer, create those mocks or that experience, put it in front of people, see how they engage with it. That can be so powerful. You could invalidate tons of hypotheses at that state. You know, you really want to only things, the only things you want to get to that kind of, that deep A-B testing environment are ones that have been kind of vetted along the way, and that way you reduce your fail rate, right? Because you're, you're failing faster by using other methods. So, you know, I think that's... I'm more advocate for that side. Like, let's fail fast by using those tools rather than figuring out a way that you can rise above every- where everyone else is operating and figure out ways to solve all that complex stuff better, because that's gonna be challenging. But you can always get better at experimenting and validating things faster.

    11. LRLenny Rachitsky40:57

        (instrumental music) This episode is brought to you by Writer. How much hype have you been hearing about generative AI? So much. But how do you take it from a shiny toy to an actual business tool that helps you do your actual job? Writer is an enterprise grade generative AI platform built specifically for the needs of businesses and already widely deployed at world-class brands like Uber, Spotify, HubSpot, and UiPath. With Writer, you can break through content bottlenecks across your organization, from marketing web pages to sales emails, in-product messages, to creating high quality on-brand content at scale. And unlike other AI applications, Writer's training happens securely on your data and your style and brand guidelines that you provide specific to your organization. The result is that you get consistent content in your brand voice at scale. Get AI that your people will love. For a limited time, listeners to Lenny's podcast can get 20% off if they go to writer.com/lenny. That's writer.com/lenny.

13. 41:58 – 49:51

    Where the best ideas come from

    1. LRLenny Rachitsky41:59

       Where do you find the best ideas come from for driving meaningful lift? Like, is it gut, instinct type and experience, uh, bucket? Or is it data telling you, like, "Hey, here's a huge opportunity." In your experience?

    2. LSLaura Schaffer42:15

       I'm a very, like, data-driven person. Like, I, I self-describe and think of myself that way. Because, you know, in large part because of that, feel really, you have to be constantly checking yourself, and data's a really great way to do that.But, uh, I definitely think that I would be described as someone who's, like, going more by their gut when looking at-

    3. LRLenny Rachitsky42:33

       Hmm.

    4. LSLaura Schaffer42:33

       ... data and results.

    5. LRLenny Rachitsky42:34

       Fascinating.

    6. LSLaura Schaffer42:35

       Just because of the way that I approach it, which is, I'm very comfortable and very common in using qualitative responses and things like that in supplement to quantitative data to make a decision, and that puts less of a burden on the quantitative to really make an assessment of whether something was working or not. You know, one of the things I see, uh, that I, I think sometimes goes against what other folks do, although I'm seeing kind of things shift a little is that 95% confidence rate, right? I, like, I, I came from, uh, my, like, my background, like, in, in college, I was in a lab running experiments or really publishing to a journal and stuff, and we had to have that 95% confidence rate. Had to because the things that were coming out of the lab and being published were influencing things like how we do education and how we understand how bias works and when it shows up and therefore how we can combat it. Things were a- were wrong and saying a bunch of baloney, like that can cause some significantly bad things like false positive, false negatives in that context can be very, um, dangerous for lack of a better word. You, you, you think of other, you know, pharmaceuticals, like, there's... The 95% confidence rate belongs in some companies and some industries because the risk of, of, you failing to impact of a false success is very high. But those of us converting users and trying to upsell folks, like, we're very fortunate to not have that level of-

    7. LRLenny Rachitsky43:55

       (laughs)

    8. LSLaura Schaffer43:55

       ... of burden on us and we can take advantage of that, right? And so there are definitely times where I will advocate for, and I will push for, and I will myself use lower con- no, lower confidence intervals than 95%, especially if that, like, doubles the amount of experiments that you can run in a year, right? End of the day, these are all methods that we use to try to validate the hypotheses that we have, and if, if, if you have, like, if you're doing a 95% confidence interval, you're still accepting a 5%, you know, some amount of, of false success. Do that a little bit more. Challenge you to do that a little bit more, and then run way more experience. If you look at the, the net of what your team is doing over the course of a year, what you're doing over the course of a year, you will be positive.

    9. LRLenny Rachitsky44:42

       Wow, that is a big idea, idea of releasing the p-value confidence interval for experimentation in data teams. Do you, like, any, everyone would be happy to, like, be excited about this, probably maybe not some data scientists on teams.

    10. LSLaura Schaffer44:56

        (laughs)

    11. LRLenny Rachitsky44:57

        Do you do that? Like, how do you act- is that how you operate on your teams? Just like, "We don't need 95% confidence?"

    12. LSLaura Schaffer45:03

        So I'll say this, this is actually very critically important. You must have this game plan set before you run something.

    13. LRLenny Rachitsky45:08

        (laughs)

    14. LSLaura Schaffer45:08

        A failure mode that I've seen so many teams fall into is they'll run the experiment or whatever it is, and then they'll u- like, make the data fit-

    15. LRLenny Rachitsky45:19

        Yeah.

    16. LSLaura Schaffer45:19

        ... the hypothesis, right?

    17. LRLenny Rachitsky45:21

        Yeah.

    18. LSLaura Schaffer45:21

        Or sometimes they'll go in without a hypothesis and just be like, "This is gonna do better things for our metrics," but not like a core reason as to why or what, you know, what exactly are we testing here. And so this, this is another area where you could absolutely fall into that trap. "Oh, it's hovering, like, at 80. I think it's good. That, that Laura person said it was cool, so I, I think that- that's fine." Like, that will always be a trap, right? So it needs to be very deliberately thought of in advance as a way, like, "Hey, here's where we're gonna validate this." And always, always, always, if you're going to, you know, accept more risk of a false success or false positive, false negative, you want to then be really thinking about how you're gonna harden your, your validation of a hypothesis. For example, let's take that one we talked about with, with, uh, Twilio, where we were kicking people out and we're sending them to that, you know, that the pil- hot dog experiment, right? And we're sending people to that experience to kind of hide the phone number. You know, in, in that case, right, let's say that we were going to accept a lower confidence interval. I would very much wanna see qualitative feedback to confirm that that hypothesis was true. I wanna be looking at the qualitative data from the ones where people were thrown into the existing flow and ones put into the docks that one of them felt more confident and more like this was really easy to get through and, like, they, they felt it was about territory and things like that, and I'd be wanting to hear from the ones who were in the other one things like, "Oh, I kinda got stuck on that phone number segment," or like, "I, like, kinda figured this out, but it just, it feels like it's I'm out of my depth." Like, I would want to be looking for other things to corroborate, um, the hard data that I'm seeing, right? And yes, it opens the door to, you know, whenever y- whenever you open the door to more risk acceptance, you are going to have some false successes there. But all of these things together can overall make it more likely that you're shipping more things that are going to positively influence the customer. And again, I can't say it enough, it is a huge risk in and of itself to not ship as much as you possibly could in a year. That is a huge risk given that very high fail rate. So, you know, I wouldn't, you know, to those data scientists, and I've, you know, chatted with a few in my time, what I try to explain is that, that article, that data, that the 80%, like, that's hard data, right, about what a detriment it can be if you are, um, you know, if, if you don't run enough experiments. You can literally, if you just run 10 in a year, odds are maybe two of them are gonna impact, two over the course of an entire year if you kinda take that approach. So, you know, data scientists can understand, "Hey, if we do this, you know, if we, we n- narrow this down, we can, like, double or triple, whatever it is, our, the number of experiments we can run, and overall net, that's gonna result in, you know, few- you know, more successes that will overall net us to a positive place." Like, you can still tell a data story to the data scientist about why you're doing this. Again, this is why, you know, when you asked that question, I identify as a very data-driven person. But I think, you know, some of the methods that I use can, uh, you know, sound at the surface level as more like, "Oh, I'm going by my gut," but again, very data-driven, it's just embracing the reality...... of some of the hard data that I don't think we all embrace or are even aware of sometimes about that, that fail rate.

    19. LRLenny Rachitsky48:42

        This is awesome. This is a big idea. Have you written about this anywhere for folks that maybe wanna try this approach at their company? And if not, you should.

    20. LSLaura Schaffer48:50

        I appreciate that. I, it's funny, I, it's been... It's like on my general life to-dos is to start writing some of this down. I have three children, one of whom is five months old, another two and four, and so, um, sometimes I'll start to write and then, like, one of them will, like, crawl across the keyboard and it's, you know, like, this... By, you know, one of them, I mean all of them, multiple times. But eventually... Yeah, I'd be very happy to, to do that if, if folks would be interested. I, um, I'm always, I'm always happy to do whatever I can to help folks, help empower folks with knowledge to do better, 'cause this, none of this is, like, secret sauce, really. It's just sort of learn from experience, and it's always better to learn from others' sh- experience than your own. It's faster. So, yeah, I would... Definitely I aspire to, Lenny. 'Cause that, I think that's the best that I can say. But eventually my kids will get older, I hear this, and, and maybe I, I can do so.

    21. LRLenny Rachitsky49:42

        Hopefully. Uh, cool. So maybe if you're watching this, YouTube, leave a comment if you want Laura to write in depth about this idea and spread it, so your company.

    22. LSLaura Schaffer49:50

        Sure.

14. 49:51 – 52:54

    Experimentation lessons

    1. LSLaura Schaffer49:51

    2. LRLenny Rachitsky49:51

       Okay, I wanna talk about growth, but I have one last question just along the lines of experimentation. Is there any other, just like, I don't know, big lessons or takeaways of just h- of running experiments that would be interesting to share?

    3. LSLaura Schaffer50:03

       I think we kinda got into this one, like, a little bit, but I just really wanna exclamation point underline it, which is... That notion of kind of making the data wrap to fit, like, a concept, you know? I think a lot of teams are, you know, feel and, and are under a lot of pressure to show progress, right? And, "What did you do this month? Where, how did metrics move?" And, like, you know, and, and it can cause folks to feel like they, they have to do that, where it's like, "Oh gosh, this experiment." Like, everyone's got this experience where you run an experiment and you're, like, looking at the data, refresh, refresh, refresh. Oh my gosh, it actually performed worse. You know? (laughs)

    4. LRLenny Rachitsky50:40

       Mm-hmm.

    5. LSLaura Schaffer50:40

       Or it's like not the same and, "Gosh, we got everyone really excited about this thing that we all, like, worked on really hard," like, "Oh my God, what are we, what are we gonna say in the QBR or, you know, the, the monthly report?" Or whatever it is that, that sort of, these thing, you know, y- the results come to, to light. Um, and, and to this I'd s- I'd say this, that it's incredibly important for growth teams to educate out and for folks outside of growth and leading growth especially t- to understand that the, the best way for a growth team s- to succeed, the, the only way really for them to succeed, is to embrace the fact that they're there to validate, to, to understand what the biggest opportunities are and to go after them. And that is not something that can be done on a weekly timeline. Sometimes even a monthly depending on the space you're in and what's known and unknown. And so any growth team that's, that's beholden to short timeline wins and improvement is always gonna be dangerous. That's a, that's an environment that's conducive to vanity metric usage and fab- you know, massaging the data to fit. And ones that are more successful are ones that are reporting over longer periods of time. 'Cause any growth team given enough time to kind of fail enough times to learn the right thing to do is absolutely gonna have, show success. Real success, right? Like, not the, "Okay, we're gonna make this data fit," but, like, real moving the metrics success. And so definitely, you know, educating out, if you find yourself in a position where you, you are beholden to that, you know, share that 80% fail rate. Like, just math, statistics, data. Like, that is not, you know, you cannot be successful in an environment, but over time y- you can be. And so, uh, that's, that's one thing I definitely would, would draw on that I spent, uh, you know, and I'm spending a decent amount of my pie chart at, uh, at Twilio and then also at, at Rapid where I was after that. And I'm sure I'll spend some time at, at Amplitude as well, just helping folks kind of understand what is the, the healthiest ecosystem, most powerful ecosystem for a growth team to operate in. And time and expectations over time is a big part of that.

    6. LRLenny Rachitsky52:46

       When you say pie chart, it's like the pie chart of your time, like, a big chunk-

    7. LSLaura Schaffer52:49

       Yeah.

    8. LRLenny Rachitsky52:49

       ... of your time goes to this. That's awesome. I like that.

    9. LSLaura Schaffer52:51

       Mm-hmm. Yeah.

    10. LRLenny Rachitsky52:51

        I use pie charts a lot as, to describe that same idea.

15. 52:54 – 54:43

    The amount of time a growth team needs to be successful

    1. LRLenny Rachitsky52:54

       Just to be a little more concrete there, what is the timeframe you think is the minimum for a growth team to be thinking a- across?

    2. LSLaura Schaffer53:01

       I think it's good, especially for, for newer teams but even teams in general, commit, commit to something that you can kind of do over the course of a year. And, you know, low, medium, high, right, is, is always helpful in that space. I mean, you know, a lot of times-

    3. LRLenny Rachitsky53:15

       What do you mean by that, by low, medium, high?

    4. LSLaura Schaffer53:16

       Like, uh, low, medium, high, more like, you know, "Hey, we've got a few bets that we have, or few core hypotheses and if they take off, that's gonna be our high bucket." Like, "Wow, like, like, we think these things could be kinda lightning in a bottle here. But they could also be, you know, a bunch of vapor mist. So... But until we run it, we're not gonna know. And if those bear out though, then yeah, that's, that's our high. And hey, we've got a few things that we think are safer. You know, um, maybe it was sort of validated a bit in the previous year or what have you, and you know, these ... metric this amount." So it's, it's helpful to, to give people though that, that, that construct, right? It, it deviates from it, very hard deviates from this notion of like, here's the single number that we're gonna hit, right? Just things that help people kind of understand that space a little bit better and what to expect. And because of that, it can be a little bit lumpy. Like, you know, there were some things that we released. I mean, it was truly for the most number of years, so it's gonna be easy to talk about this, this construct here but, like, you know, there's one thing that we did that generated, like, tens of millions of dollars in the pipeline. It was really, really powerful and took, you know, some time to kind of navigate and, and, and validate. Other times we did, like, that onboarding stuff, I was talking about like, kind of like catching those things, like, that was, you know... It could happen on a little bit of a, a faster clip but still, you know, took some time to, to validate and understand. But yeah, over the close- course of a year, you should generally kind of be able to commit to, to movement but help people understand the methods there so that they're not coming at you on a weekly basis being like, "And what did you do this- these past couple

16. 54:43 – 58:41

    The big change at Twilio that led to tens of millions of dollars

    1. LSLaura Schaffer54:43

       days?"

    2. LRLenny Rachitsky54:43

       Okay. I gotta, I gotta follow up on a couple of these things. Uh, what is it that... What was that big change at Twilio that led to tens of millions of dollars?

    3. LSLaura Schaffer54:50

       This is part of, um, the, the, the course that I teach at Reforge, so, um-

    4. LRLenny Rachitsky54:54

       Oh, amazing. Good plug for Reforge. What's the course called? Just so folks-

    5. LSLaura Schaffer54:58

       Um-

    6. LRLenny Rachitsky54:58

       ... want to check it out.

    7. LSLaura Schaffer54:58

       ... ............................ retention. I think it's, um, live-

    8. LRLenny Rachitsky55:01

       Awesome.

    9. LSLaura Schaffer55:01

       ... is, is my part. I think it's, um, like right now we can... I'll ge- I'll give you a link so we can-

    10. LRLenny Rachitsky55:04

        Yeah, we'll link to it.

    11. LSLaura Schaffer55:05

        ... put it in, in the show notes. But, um, yeah. The high, high level w- version is we... You know, this was, this was kind of deeper into my, my journey at Twilio. This is, uh, you know, fast forward a, a few years, you know, to kind of build up this, this team and some cool things going on. But I was really looking for kind of what's, like, the next big thing for us to do. Like, what, what could that be? And I noticed, you know, remember I was, you know, that the question very back in the day when I asked about like, the kind of developer versus not developer folks in our-

    12. LRLenny Rachitsky55:34

        Mm. Yeah.

    13. LSLaura Schaffer55:34

        ... program. We saw that that little non-developer little doodad was kind of growing, right? We were actually the number of people in the ecosystem who were identifying themselves as not a developer we're, we're in the space. But very interestingly, they were, as we kind of got, got more refined in our understanding of those folks, a lot of them wanted to build with Twilio. Like, it, you know, there was a hypothesis of like, "Oh, well maybe they're lost. Maybe they just want pricing. Like, maybe they signed up by mistake." And I was like, "Nope. They're here to build. They wanna build." You know, and they, they kind of struggled through the developer onboarding and some of them would succeed and some wouldn't. But anyway, um, the... It was all about identifying, like, what did they need to succeed? Like, were they... If we were s- made them successful, could it contribute to dollars? You know, one of the core learnings I'd heard from sales at the time was, "Hey, it's very challenging for us to get the kind of folks w- where, when a developer's not involved yet to, to go from zero to one to get something off the ground. But, man, if we can get them to do that, if I can get them to $1 in spend, I can get them to five. And then five, we can get them to 50. Th- th- like, 10,000. Then I can get them to 100,000." This whole long journey, like, "Hey, Laura, if your team can just get them, like, off the ground, man, we can do so much." So yeah. The, y- journey was all about, okay, what, what were the things that were missing in the experience we were offering? And, you know, ultimately was they couldn't write code from scratch, that was really difficult, and also weren't gonna stand up a server. That was difficult. But we ended up, um, iteratively experimenting a way to, to validate those hypotheses and what's the right way to do this. And yeah, that was great. It's called, uh, Quick Deploy on Code Exchange. Um, anyone can go there and deploy an app without having to write code and kind of get an aha moment there with Twilio.

    14. LRLenny Rachitsky57:16

        That is awesome. So basically it's like a low-code Twilio app.

    15. LSLaura Schaffer57:20

        Yeah. It ended up kind of being like... We had a lot of, like, like pet names, like nicknames for it. I think probably the one that, that most succinctly describes it is just it ended up being kind of a create your own demo experience, which, you know, made-

    16. LRLenny Rachitsky57:31

        Wow.

    17. LSLaura Schaffer57:32

        ... made, you know, you talk about the psyche of people, you know? You know, we talked about how developers, telecom, and Twilio can be inhibiting. Well, talk about like this, you know, the, the non-developer, like sometimes the buyers or like the people who are, you know instantly buying, buying decisions, like for them it was like, not only was it like telco, but it was like the developer stuff was inaccessible. But they still wanted to, you know, jump in and they wanted to have that experience. And so this was a way for us to give them momentum, give them confidence. "Well, geez, if I can get this running my development team, they'll definitely do it." Right? And so it was a very powerful kind of, uh, uh, you know, moment where we could really address the psyche of those users, get them excited about Twilio, and then give sales kind of the ability to give something powerful to those non-engineering buyers and folks they're talking to.

    18. LRLenny Rachitsky58:17

        So genius, looking back. Seems like an obvious win. I imagine it was not. One of my readers suggested that I start a series of the story of a feature and kinda walk through the-

    19. LSLaura Schaffer58:26

        Mm-hmm.

    20. LRLenny Rachitsky58:27

        ... discovery, ideation, development, iteration. And this feels like a really interesting example of that. But anyway, uh-

    21. LSLaura Schaffer58:33

        Cool.

    22. LRLenny Rachitsky58:33

        ... I got just a couple more questions. I know we've been going for an hour now, so I-

    23. LSLaura Schaffer58:36

        (laughs)

    24. LRLenny Rachitsky58:36

        But I, I have questions. I don't wanna let you go just yet.

    25. LSLaura Schaffer58:39

        Yeah. That's fine.

    26. LRLenny Rachitsky58:40

        And they're, they're on

17. 58:41 – 1:05:42

    The need for both PLG and enterprise, and how Amplitude plans to tap into PLG

    1. LRLenny Rachitsky58:41

       growth. So, so one question is just, you worked at Twilio, which is very product-led growth. You're gonna be... You're now at Amplitude, which is-

    2. LSLaura Schaffer58:49

       Mm-hmm.

    3. LRLenny Rachitsky58:49

       ... more sales driven and I know it's... you're trying to go more product-led.

    4. LSLaura Schaffer58:53

       Yeah.

    5. LRLenny Rachitsky58:53

       I know Elena talks a lot about this, how every company needs to have product-led motions, otherwise they're gonna be disrupted by someone that becomes product-led.

    6. LSLaura Schaffer59:00

       Totally.

    7. LRLenny Rachitsky59:01

       And I don't know what Hire is. Which, which bucket would they fall into?

    8. LSLaura Schaffer59:05

       Yeah. Between the like, like SLG and PLG. I mean, yeah. For, for me, it's, um... They're, they're kind of two sides of the same toin- coin. Like, you know, growth, part of growth and sales, it's all, it's all, to me, very thematically the same stuff. The difference is that with growth you are selling with your product, and with sales you're selling with person, like one to one. And, you know, so, so companies need to be, you know, employing both of those forces to optimally convert their audience. Um, you know, we're in a world where people are expecting both. They're expecting to be sold by your product and sold at the enterprise level in large companies by, by a human being that's gonna listen s- to their specific needs and really break it out for them. And, you know, if you only have one, you're gonna, you're gonna miss stuff. So absolutely, I think y- you wanna... Tho- those two forces together working well. And obviously there's sort of, you know, different stages, you know, things work differently in different spaces. But, you know, I, I think when it comes to, to Amplitude, I think there's a huge, you know, opportunity here. I think the key is and the, the, the challenge, you know, for companies that have done like the sales thing and are trying to crack into the PLG thing, is, you know, really comes down to how you fundamentally are, are approaching that space and again, your users and, you know, the, the where they're at and the psyche of where they're at. I think a lot of companies will say, "Well, okay. Like, hey, we're gonna do this PLG stuff. Let's take that sales, like, enterprise whatever offering that we have, and let's, you know, chop it up a bit and, like, cut excess here and, like, ch- cut out this feature here, and we're gonna, like, slap this plan out and we're gonna put a price on it. And we'll maybe have, like, hours of debates over whether it's like 10.99 or, like, $104 or, like, 75. And eventually we'll, like, someone will win that battle and, like, slap it on and then kinda see what happens." And anyway, the discussion or the focus is a lot around the product. "What are we gonna do with this product? How are we gonna crack it open and shift it?"... and, and then give it to these, you know, people, these users, these, these visitors. And what it's missing, I think, is... And a lot of times it's easy to miss, is that when we're doing PLG and we're shifting from sales to PLG, we need to reset. We need to recognize that, you know, again, this is, this is sales. Sales via the product. What do you do when you're... What does a good sales rep do when they're engaging? They, they understand what the problem is of the person in the space they're talking to, so we need the same thing here. What are the unique problems of people who are coming into our self-service space? And I think, you know, when it comes to a company like Amplitude, you know, you're, uh... You know, a lot of the, the folks that we'll be kind of looking to address via the, the PLG motion, you know, there's a number of things we wanna achieve there, but one of the primary things is to kind of tap into this and view market and really give them a really... You know, start-ups and give them a space to land and to grow. And, you know, you have to think like, "What are the challenges and unique problems that they have?" 'Cause we're gonna be using our product to kind of sell to them. We need to meet them where they're at with the problems that they've got. And I think one of the things that is... That I've, I've observed from being in, you know, all these start-ups and, you know, advising some start-ups is, I, I very rarely... I don't think I've ever come across a start-up where they have, like, the right number of analysts for their needs. In fact, a lot of them, like, don't have any. And so what that means is that the CEO is, you know, being an analyst to create their dashboards for the board. And, you know, the product manager is, like, being an analyst to figure out, like, what the heck is going on and creating reports for their product. And that's happening all over the place is that people are in their roles and they have to be analysts too. And I think that that's a problem that especially, you know, younger companies and early stage companies have. And so when they're, you know, coming back to their psyche, what are they caring about? What are they thinking about when they're signing up for product analytic product or something? They're looking for something that's gonna help them feel reassured that they're gonna be able to actually get to the bottom of the right metrics, create the reports that show things the right way. I... You know, what's the best way to show churn? There's gotta be a best way, like so many people are doing it. Guess what? There are some really good ways to do it, and there are some really successful, you know, ways to, to set up dashboards for the board. People have done that too. Like, there's a lot of that knowledge that exists and a lot of those templates and frameworks that exist. Benchmarking. Are these numbers even good, right? And so, one of the hypotheses that I have is that, you know, if we, we take that perspective and we understand, you know, that that is the problem, that there's a number of things that we can do to really change the way that self-serve experience works to help convert people and show them how Amplitude can make them kind of that powerful. But, but the thing that I think sticks across all companies, not just Amplitude making the shift, is just that, that when you're doing this, do not think this is a copy-paste but, like, chop it for parts thing. Don't start with your product when you're building out your strategy. Start with your customers, your users, your prospects, your... The people who are gonna be coming into your self-serve flow. Make sure you're understanding how their problems differ, because they do, from the people that you're addressing at the sales-led side. And then make sure that you're orienting your experience and product around those people.

    9. LRLenny Rachitsky1:04:12

       It's interesting that you almost have to kind of start again as a... as a product company, as a product... Yeah, because you may need to solve completely different problems that eventually lead to the same place. But it's interesting what you're saying, that you may end up targeting, like, analysts or PMs or... I know Amplitude r- has always focused on PMs, but...

    10. LSLaura Schaffer1:04:32

        Yeah, and it's right. And there, there is always... The nice thing about it is it's... In some ways, it does feel like you're starting fresh, because you, you do need to kind of start with the customer again and, like, what's their problem. But in a lot of ways, you know, you, you can carry over a lot of the same knowledge. I mean, at that point you kind of know what's working well. Like, Amplitude, for example, does have a ton of knowledge around what some of the best ways are to set up reports. Like, there's a lot of things that they have the momentum going, sort of like where do you choose that momentum and how do you put that and curate that in front of users and make sure that they're getting the right things? But there's a ton of momentum already there, it's just a little bit about harnessing it, understanding that, yeah, where are the gaps? 'Cause there are gonna be gaps. But, you know, anchoring in a customer problem is, I think, the way that you start any new product. Any new thing that you're releasing should always think about, like, the customer and the pain point. So no different than when you're doing, you know, PLG for the first time or cracking into it. You need to be thinking again, starting again with the problem, the problems they have, the psyche that they have in coming to your, your space, so that you can build something that is gonna effectively make them feel like, "Oh, you can solve my problem. You get me." And, and show them how your product's gonna do that.

18. 1:05:42 – 1:11:16

    What it’s like to serve developers

    1. LSLaura Schaffer1:05:42

    2. LRLenny Rachitsky1:05:42

       Final question, and this is around developers. You worked at Twilio. Obviously Twilio sold to developers.

    3. LSLaura Schaffer1:05:48

       Yeah.

    4. LRLenny Rachitsky1:05:49

       I think Rapid, where you worked-

    5. LSLaura Schaffer1:05:50

       Mm-hmm.

    6. LRLenny Rachitsky1:05:50

       ... right before Amplitude, also sold to developers. Selling to developers feels like such a hot space right now. There's so many start-ups that are just building developer tools, such a huge market. Used to be not. Used to be like, "There is not a market in developers." They don't... They're not gonna spend money. There's not enough of them. And now it is a big popular spot. And so I'm curious, what have you learned about building a start-up and a product that sells to developers? Uh, I imagine a lot of founders building sorts of tools. It'd be really curious.

    7. LSLaura Schaffer1:06:16

       The first is that developers are just a very different audience from any others. I've seen so many people who have come in strong, done growth really well or product really well with other audiences, and they're like, "Oh, like, I'm gonna take all those learnings and I'm gonna pivot into serving developers." It ends up being a very steep climb, because developers are so different. Um, and let me give you, like, a couple just fun facts-

    8. LRLenny Rachitsky1:06:39

       Yeah.

    9. LSLaura Schaffer1:06:39

       ... that make them really different, and some of these have kind of some interesting stories. One is developers, like, almost two-to-one do not look at your marketing website at all. They go straight to your sign-up flow. So what that means is, all that beautiful context that you're setting and, you know, product data, pricing, all that stuff, like very, you know, very often, they're skipping all of it context-free and going straight to your sign-up.And so anything that is not the... You know, any time you make an assumption like, "Oh, well, they probably know this coming into sign-up," or like, "Well, we don't need to include... That's on the marketing website," like, none of that's gonna apply to this group of people. They are like, they're there. They're, they're looking at you. I, um, I describe them, like the analogy I have for, for this group is like they're the, the IKEA buyers who, you know, when the IKEA package comes, they're not opening up the instruction manual and reading it-

    10. LRLenny Rachitsky1:07:30

        Mm-hmm.

    11. LSLaura Schaffer1:07:30

        ... and then starting to go through. They're in there tearing open the bags and starting to, like, pull the pieces together and trying to build it, right? They'll come up for context and, and, and steps and such when, when they get stuck if they're motivated. So, uh, that's, that's one thing. And then, you know, another one is just the aversion to talking to sales. And I think everybody can... You know, hearing that sometimes like, "Oh, yeah. Well, I hate sales too. Man, I, when I sign on and get bombarded by sales, that's the worst." I totally get that. But I've... Just developers are on this whole other level. There was a, there was a FANG company sign up for Twilio, built a POC, launched to production, all this, and, and operated in that space for months without engaging once with sales. I was trying to reach them, right? And I, I ended up being the one that talked to them, like, first, because they, like, reached out to support because there was something about their delivery that was, was, was off there, like s- like missing a feature, and they did not want to talk to sales. They ended up talking to me.

    12. LRLenny Rachitsky1:08:27

        Hmm.

    13. LSLaura Schaffer1:08:27

        And this was when I was in product marketing, and that was, like, my first exposure of, like, these people do not want to talk to sales. You know, and then there's another one where, like, a giant retail company where, like, the engineering team signed up with their personal email addresses so that they wouldn't get bombarded by, by sales. It was only, like, later that we, like, found out.

    14. LRLenny Rachitsky1:08:44

        Hmm.

    15. LSLaura Schaffer1:08:44

        Well, anyway, it's... So, but, but the thing that's most important, these are fun facts, but, but the thing that is, uh, I would say is the most important to leave, thing to leave with, with listeners here is, why, like, what makes them s- so different. Like, like why? Like, why, what's the deal here? And it stems from their charter and their responsibility. So, you know, if we put ourselves in developer shoes for a minute, a developer, if a developer is required to use your product, especially if they're like the primary user, the primary builder, it's really important to recognize that they are responsible for that. If your service goes down, that's their responsibility, not just for themselves, but their team. If the pager wakes up someone because the, the service they bought from you goes down, that's on them. If, oh, turns out that, uh, it doesn't work with, with the systems that, uh, they said it was, well, that's on them. Like, doesn't integrate with the data the way that, you know, everyone wanted it to, that's on them. Everyone looks at a developer when it's not working right, and it can work right in so many ways. That's their failure. That can, it can cost them their job. It could cost them the trust of their team, cost them their reputation, you know? And that means that the stakes are very high for them every time that they're adopting something new. So they can't afford to take someone's word for it, especially a sales rep who might be, have some other motivations, right? From their perspective, right? Like, they, they can't afford to trust your content or someone's word. They must do it. They must prove it to themself. And so that's why they, for developers to be bought in, they need to do something, build something, a proof of concept at the very least, if not kind of like moving further than that. And so that means they're gonna be pretty darn deep in their self-serve experience with you before they're ready to commit. And so if, if you're a company that is providing, uh, that, that requires developers to build, right, you must invest in self-serve experiences in order to effectively convert your audience, and you should be thinking of them as something akin, your, your self-serve function a- and growth folks, someone akin to a sales force, because your sales, developers are not gonna accept sales coming in and trying to convert them at that stage.

Episode duration: 1:21:12