Sean Ellis: Why activation moves retention more than nudges

Through the very-disappointed survey and must-have user research; Lookout moved from 7% to 60% by repositioning, and activation now drives real retention.

Lenny RachitskyhostSean Ellisguest

Sep 5, 20241h 44mWatch on YouTube ↗

EVERY SPOKEN WORD

150 min read · 30,141 words

0:00 – 2:18
Sean’s background
1. LRLenny Rachitsky
  ... the Sean Ellis test, such a seemingly simple idea that has had such a profound impact on the startup world.
2. SESean Ellis
  The question is, how would you feel if you could no longer use this product? Once you got a high enough percentage of users saying they'd be very disappointed, most of those products did pretty well. If you fell too low, those products tended to suffer.
3. LRLenny Rachitsky
  Say someone is listening and they're like, "Okay, man, I'm getting like 10%. I don't know what to do." What do you find often works?
4. SESean Ellis
  Just ignore the people who say they'd be somewhat disappointed. They're telling you it's a nice-to-have. If you start paying attention to what your somewhat disappointed users are telling you and then you start tweaking onboarding and product based on their feedback, maybe you're going to dilute it for your must-have users.
5. LRLenny Rachitsky
  Moving retention often is really hard, but I guess it sounds like there's often something you can do.
6. SESean Ellis
  It's usually much more a function of onboarding to the right user experience than it is about the kind of tactical things that people try to do to improve retention.
7. LRLenny Rachitsky
  What are, like three or four things that you think people should definitely try to help improve activation?
8. SESean Ellis
  In my experience...
9. LRLenny Rachitsky
  Today, my guest is Sean Ellis. Sean is one of the earliest and most influential thinkers and operators in the world of growth. He coined the term growth hacking, invented the ICE prioritization framework, was one of the earliest people to use freemium as a growth strategy, and maybe most famously developed the Sean Ellis test to help you understand if you have product-market fit, which a large percentage of founders use today and profoundly impacted the way startups are built. Over the course of his career, Sean was head of growth at Dropbox and Eventbrite, helped companies like Microsoft and Nubank refine their growth strategy, was on the founding team of LogMeIn, which eventually sold for over four billion dollars, and he's the author of one of the most popular growth books of all time called Hacking Growth. In our conversation, we dive deep into two topics. One, how to know if you've got product-market fit and what to do if you don't. And two, how to figure out how to grow once you found product-market fit. If you're in the early stages of a new product wrangling with product-market fit or trying to figure out how to jumpstart or further accelerate growth for your product, this episode is for you. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It's the best way to avoid missing future episodes and it helps the podcast tremendously. With that, I bring
2:18 – 6:28
The Sean Ellis test explained
1. LRLenny Rachitsky
  you Sean Ellis. Sean, thank you so much for being here and welcome to the podcast.
2. SESean Ellis
  Thanks, Lenny. I'm, I'm super excited to be on with you.
3. LRLenny Rachitsky
  There's so much that I want to talk about, there's so many directions we can go. But to keep it focused, I want to spend time on two areas. I want to talk about how to know if you have product-market fit.
4. SESean Ellis
  Mm-hmm.
5. LRLenny Rachitsky
  And what to do once you have product-market fit in terms of figuring out how to grow.
6. SESean Ellis
  Yeah.
7. LRLenny Rachitsky
  And I know these things are very linked. I know you spent a lot of time on these things. How does this feel?
8. SESean Ellis
  Sounds perfect. Yeah, let's do it.
9. LRLenny Rachitsky
  Okay. Okay, amazing. Let's talk about, first of all, the Sean Ellis test/something people call sometimes the product-market fit test. Such a seemingly simple idea that has had such a profound impact on the startup world. I've never actually seen you talk about the history of this thing, how you came up with these questions, how you came up with 40%, the whole journey of this thing. So let's talk about this. But first of all, can you just tell people what is the Sean Ellis test for folks that aren't exactly familiar with this?
10. SESean Ellis
  It's a simple question that helps you figure out, you know, does anyone consider your product a must-have or, you know, ideally who and how many people consider it. But, but ultimately, it's about trying to figure out, you know, do... Is your product a must-have, which could be equated to, to having product-market fit. And so the question is, how would you feel if you could no longer use this product? And I give them the choice, very disappointed, somewhat disappointed, or, uh, even not disappointed, or not applicable, I've already stopped using the product. And what I'm trying to find are, are those people who say, "I would be very disappointed if I could no longer use this product." And that's, that's a really powerful vein to dig into when you, when you discover that you actually have some people who would, who would give a crap if your product disappears. (laughs)
11. LRLenny Rachitsky
  This episode is brought to you by Gamma, an entirely new way to present your ideas powered by AI. If you hate designing slides and dread that feeling of staring at a blank slide, Gamma is here to help. Just upload your PRD and turn it into a beautiful, ready-to-present presentation in seconds. Gamma works with all types of formats, from Google Docs, PDFs, to PowerPoint. You can even drop in a link to your favorite Lenny's newsletter post and turn it into a presentation for your team. Gamma has become one of the fastest growing AI web products in the world, adding 20 million new users just this past year and is setting its sights on becoming the modern alternative to PowerPoint. Whether you have design skills or not, Gamma can save you hours of time synthesizing your ideas and shaping your content. Visit gamma.app and use promo code Lenny to get a free month of Gamma Pro. That's G-A-M-M-A dot app. Let me tell you about Commandbar. If you're like me and most users I've built product for, you probably find those little in-product pop-ups really annoying. "Want to take a tour?" "Check out this new feature." And these pop-ups are becoming less and less effective since most users don't read what they say. They just want to close them as soon as possible. But every product builder knows that users need help to learn the ins and outs of your product. We use so many products every day and we can't possibly know the ins and outs of every one. Commandbar is an AI-powered toolkit for product, growth, marketing, and customer teams to help users get the most out of your product without annoying them. They use AI to get closer to user intent, so they have search and chat products that let users describe what they're trying to do in their own words and then see personalized results, like customer walkthroughs or actions. And they do pop-ups too, but their nudges are based on in-product behaviors like confusion or intent classification, which makes them much less annoying and much more impactful. This works for web apps, mobile apps, and websites. And they work with industry-leading companies like Gusto, Freshworks, HashiCorp, and LaunchDarkly.Over 15 million end users have interacted with CommandBar. To try out CommandBar, you can sign up at commandbar.com/lenny and you can unlock an extra 1,000 AI responses per month for any plan. That's commandbar.com/lenny.
6:28 – 8:06
The 40% rule
1. LRLenny Rachitsky
  And the idea is that if you... 40- 40% or more of people say they'd be very disappointed if they could no longer use the product, you essentially have product market fit.
2. SESean Ellis
  I would say it's a leading indicator of product market fit. The, the, the lagging indicator is do they actually keep using it. So probably retention cohorts are, are more, more accurate, but the problem is, like if you, you know, like your time at Airbnb when (laughs) ... How long do you have to look at a retention cohort before you know that you've actually long-term retained someone? And so with this question, you can, you can kind of find out day one. You don't actually even need to be... You don't need a, a good analytic system in place to be able to see if, if product market fit exists. And so, um, yeah, the- the 40%, the 40% was not something I originally had in there. I was, uh, also originally I was, I was trying to have just a filter so that I was not treating all feedback from customers the same, but I was, I was trying to find feedback from, from customers who actually really cared about the product, and then, uh, it was over time as I, uh... Yeah, at the time, I was, I was working for a couple of YC-backed companies, and so those companies were all pretty, pretty connected, and, and so I would share, share the question with a lot of other startups in, in Silicon Valley. And so over time, I started to see there was a pattern that, you know, once you got a high enough percentage of users saying they'd be very disappointed, most of those w- very disappointed without the product, most of those products did pretty well, and then if you, if you fell too low, those products tended to suffer.
3. LRLenny Rachitsky
  Okay. There's two things I want to
8:06 – 12:34
Case study: improving product-market fit
1. LRLenny Rachitsky
  definitely follow up on here. The first is such an important point that you made at the beginning when I introduced this test, is that you describe it as a leading indicator of product market fit and actually retention. People actually using your product, the product actually-
2. SESean Ellis
  Mm-hmm.
3. LRLenny Rachitsky
  ... being used by the market is the actual ultimate test.
4. SESean Ellis
  Right.
5. LRLenny Rachitsky
  So the idea here is this is a good way to get a sense of before you actually have data, are we headed in a good direction? Can you just speak more about that, like how to... when to use this and when it's most useful in this scenario?
6. SESean Ellis
  Yeah. And so for me, in particular, when I come into a company, my, my, my goal is to help them grow, and so I don't want to put myself in a situation where I'm going to fail because no one actually cares about the product. And so it c- it can really be asked at- at a company at any stage. It's- it's helpful to understand who your must-have users are. But essentially once- once you have an- uh, an- even an MVP of it, like a- a very first MVP on the product, you can still get some- some useful feedback about the product if- if- if it's resonating with anyone. So I actually had a- a company where, um, I a- I had committed to work with them. It was right after I- I left Dropbox and I, I'd committed to work with these guys for six months to help them grow. I ran the question and it came back at only 7% of users saying they'd be very disappointed a- with- without the product. And so I'm like, "I have six months to help them grow and they're only at 7% right now. Who kn- it might take six months to get to 40%. Am I doing them a disservice by being in a growth role and- and, you know, being on payroll during, during this period of time?" But fortunately with the signal and the information we got from the initial survey, we were able to get them at 40% in two weeks.
7. LRLenny Rachitsky
  Wow. What did you do there? Just as a-
8. SESean Ellis
  (laughs)
9. LRLenny Rachitsky
  ... case study?
10. SESean Ellis
  Yeah. Yeah. So the company called- called Lookout. It's a mobile security company and, you know, now most of the things in Lookout were- are- are, like, built into iPhones and Androids, but at that time the product had, uh, had everything from, like, back up my data, to find my lost phone, to protecting your phone from, uh, you know, w- with f- with firewall and antivirus. And so when we ran this initial survey, I dug into the 7% who said they'd be very disappointed without the product and found that, um, most of that 7% were focused on the antivirus functionality. So they were like, they know they need to protect their- their computer from viruses, smartphones were becoming more like, uh, computers, so it just made a lot of sense for them to... that they'd need to protect their- their phone. And in- interesting, at that time, I think there was only, like, one kind of phone virus that had ever even happened, but it was a- a pretty easy mental leap for people. And so now we knew, okay, it's antivirus that people really valued, and so step one was just, uh, reposition the product on antivirus. So that- that kind of creates a filter, so anyone who now is coming in to sign up for the product who doesn't care about antivirus is not going to convert, and those who are excited about antivirus are going to convert. We already know from the initial survey that- that people value that after they convert, so by setting the right expectations around it up front, you're gonna bring people in with the right expectations. But then the second thing that we did was we streamlined onboarding so that the first thing that they did after signing up for the product was to set up the antivirus and then get a message, "You're now protected from- from viruses." And so it's really the combination of those two things. It's set the right expectations and then speed the value. And so the next cohort of people that we surveyed were at 40% saying they'd be very disappointed without the products. That was... Literally it took two weeks to make those changes. Six months later, it was 60% on- on the- on the score, and then, uh, I think they hit the- the billion dollar valuation four or five years later on, uh, you know, ultimately being one of the, one of the, you know, early unicorns. And interestingly, they... You know, as all of those things were built into mobile phones now, they- they've completely changed their business, but they continue to do really well. Um, but they've continued to iterate the business. I think that having...Having that kind of finger on the pulse early in the business was, was important to, to build the muscle in the business to be really, uh, responsive as the market changed.
12:34 – 16:50
Understanding and leveraging customer feedback
1. SESean Ellis
2. LRLenny Rachitsky
  Shawn, this is already amazing. There's just like a fractal of topics I want to explore from this very short conversation already. (laughs) So, the first is just following this thread of basically you're sharing kind of a growth strategy that I imagine you execute is look for the percentage of people that would be very disappointed if your product went away, see who they are, see what they're excited about, and lean into that both positioning wise, onboarding wise, and probably also cut out stuff from your product that they don't care about.
3. SESean Ellis
  Yeah. And I, and I was coming at it from a, a marketing perspective initially, you know? Over time, I, I positioned myself more in, in a growth role with product and, and marketing, uh, as, as areas I could influence. But as a marketer, I, I, I probably didn't have a lot of influence on a, on a, uh, engineering founded company to say, "Let's cut, cut out stuff." So, uh, it made more sense to say, "Let's just sequence the onboarding so that we're, we're highlighting this and, and onboarding to this." That was a little easier to sell.
4. LRLenny Rachitsky
  And just hearing that you can move this score so quickly without a l- even changing the product substantially, uh, I imagine would surprise a lot of people when you think about, like, re- moving retention often is really hard. And maybe we talk about that, but I guess it sounds like there's often something you can do that's not very hard that might significantly shift, uh, this product market fit test.
5. SESean Ellis
  Right.
6. LRLenny Rachitsky
  Uh...
7. SESean Ellis
  And then ultimately, like, yeah, moving retention is really hard, but it's, it's usually much more a function of onboarding to the right user experience than it is about the kind of the tactical things that people try to do to improve retention.
8. LRLenny Rachitsky
  Okay, I wanna put a pin in that and come back to that, 'cause that's a really important topic. I wanna come back to, say someone runs the survey, and they get 40%. What should they have in their mind of like, "This is what this is telling me"? Because I think a lot of people are like, "I got product market fit. I got this. Let's go, go, go." What's the best way to think about what this tells you?
9. SESean Ellis
  Yeah. I mean, it, it does, it tells you something really important, which is, you know, you're, you, you haven't created something that, uh, people don't care about. So, that's, that's im- an important insight, but until you deeply understand that product market fit, you, you kind of don't have the tools to be able to grow the business. So, that's, that's really the next step is to, to dig in and figure out, you know, who considers it a must have? How are they using the product? What did they, what did they use before? What problem are they solving? Um, one of my, one of my favorite questions is, uh... So I, I, I tend to have a lot of questions that I, that I build off of that, that I'm using that filter of, you know, trying to drill into the, the users who say they'd be very disappointed without the product. And o- one of my favorite questions is, "What is the primary benefit that you get?" And then, um, I use that initially as an open-ended question to k- to kind of crowdsource different benefits people are getting, but then, then I run another survey where I turn it into a multiple choice question, force them to pick one of four kind of distinctive benefit statements, and then the question that follows on that next survey is, "Why is that benefit important to you?" And then I start to get really good context. So, the... I actually came up with this question when I was working with, uh, an early YC company called, um, Xobni, and, uh, which is inbox spelled backwards. And I, when I, when I ran that question, basically the people who said they'd be very disappointed without the product were focused on, "Xobni helps me find things faster in my email." So it's, it's great to know, okay, that's the benefit, but when I asked, "Why is that benefit important to you?" they said, "Ah, I'm drowning in email." Like, I just, I kept seeing that statement as, as a written statement. And so when I then was trying to, to, to figure out how to acquire customers, when I tested, "Drowning in email?" that set such a good hook that was the context that people were living in, that they were really responsive to the message of, "Find things faster with Xobni," and then a description of what Xobni is. So, like, uh, I think when you can, when you can really dig into the context of why that must have benefit is important to people, you, you start to, you start to get the ingredients to build that flywheel that leads to long-term sustainable
16:50 – 22:22
Challenges and nuances of product-market fit
1. SESean Ellis
  growth.
2. LRLenny Rachitsky
  So what I'm hearing is whether you have 40%, whether you have 60% or even 7%, the actual best use of this tool is look at that percentage of highly disappointed and see what they're looking for, what they're excited about-
3. SESean Ellis
  Just start drilling in, start peeling back-
4. LRLenny Rachitsky
  ... start drilling in.
5. SESean Ellis
  ... that onion and just deeply understand them and, and, uh, make sure that you're, that you're, you know, ultimately your road- your product roadmap is doubling down on the things that are important to your must have customers, your onboarding is bringing new people to the right experience, your messaging is setting the right expectations, your, your acquisition campaigns are, are targeting people who actually have the need. (laughs) And so it's, it's all about getting the right people to the right experience and then even your engagement loop is about just reinforcing what, uh, you know, how to get people to experience that benefit more often.
6. LRLenny Rachitsky
  Awesome. And the 40% threshold, so what you shared is you basically emerged from just looking at tons of startups, doing the survey, and finding a pattern. How firm is that 40%? Like how big of a deal is it, 39 versus 41?
7. SESean Ellis
  I don't think it's that firm. You know, to me, I think the real power is having some kind of, some kind of target for the team to be shooting for that basically says, "We're not gonna aggressively start to grow until we hit this target." And I think that as just a f- focusing, um, piece is really, really important because I think one of the, one of the biggest challenges in, in an early stage startup is half the people feel like, you know, we're, we're years from, from having this product ready to grow and half the people are like, "What are we waiting for?" Where if you can actually get people on the same page of what is...Wha- what does product market fit look like for our business and, and it's at that point that, that we're gonna... Yeah, and I, and I, before I ever heard the term product market fit, I, I remember the conversations back at, at Log Me In in the mid-2000s of, of kind of like when do we step on the gas? What is, what is the combination of factors that need to be in place before we start pouring fuel on the, on the early fire? And so, yeah, I, I think that kind of, you know, nail it then scale it, it's probably been a, a, a term that's been around for decades now. But you know, it's a- it's all kind of pointing to that same concept of product market fit.
8. LRLenny Rachitsky
  How often do you, do you, have you seen false positives with this test where someone gets 40% and something is not a- not right, they, uh, they're actually far from it? Or is it generally pretty accurate?
9. SESean Ellis
  If, if you're having people say that they'd be very disappointed without your product, that, um, that's, that's a really good sign. What I can tell you is that not, not just a, not necessarily a false positive, but, like, what is driving people to say they'd be very disappointed? One of, one of my favorite books is, uh, Hooked by Nir Eyal, and he talks about in the, in the kind of engagement loop that your last step is investment. And so I ran the survey on a business that I thought was a fairly commoditized business and, um, I wanted, part of it I wanted to see could I, could I use the same go-to-market approach on a later stage company and, and, and use it to accelerate growth. And so this was a, a business called, um, Webs.com and, uh, they, they eventually got acquired by Vistaprint, but they'd been, they'd been a pretty flat for the year before I went in there, and then I, I started to kind of use this approach to, to try to dial in, uh, their, their, their growth engine. And I ran the survey thinking, you know, you've, you've had products like Wix and Weebly that have, have come onto the market since this, you know, more legacy website building product has been around, and I personally think they're a little easier to use. They're a little better. And so I didn't have high hopes when I ran the survey, but it came back with one of the highest scores I'd ever seen and it was like, like 90% of the people saying they'd be very disappointed if they could no longer use the product.
10. LRLenny Rachitsky
  Holy shit. I've never seen that.
11. SESean Ellis
  And I was like, "How could that be possible?" This, this product is, it's kind of a commoditized category. I wouldn't even say it's one of the best. And, and then when I went and dug into it, again, it comes back to that Nir Eyal Hooked model is that the investment people have made in building that website, they put so much into the, you know, that they, they know exactly how to make the changes in, in the kind of the, the, uh, CMS kind of side of things. They have, have spent a lot of time just making it beautiful. And so, so ultimately, um, it was something that that, that was why they were saying they'd be very disappointed. I mean, but, but for, you know, kind of fast-forward and, uh, on when I initially went in, still doing these things helped the business resume growth and, and, and have significant growth over the next 12 months after we did these things. So still, the signal we got from why people would be very disappointed, uh, without the product wa- was important and speed to value and kind of all, all of the, all the other things I think about in go-to-market for an early stage product still were relevant, but just the, I think they're, they were a little stronger on the, uh, you know, percentage you'd be very disappointed. Even, even Eventbrite when I was there, when we ran it was probably the second highest I'd ever seen. But with event organizers, if they've already set their event up on that platform and they've sent it out and, and to, to their list and, and all those people are coming in and they're managing their event, again, they've, they've invested a, a lot in the platform. So, so sort of switching costs I th- I think can, can factor in there. So it's, it's a, a function of both switching costs and utility of the product.
22:22 – 23:46
When to use the Sean Ellis Test
1. SESean Ellis
2. LRLenny Rachitsky
  So that, that's a question I wanted to ask is what's, what's your guidance on when to ask this question? What I'm hearing is if you ask it very far along the journey when they're very invested, uh, you'll get a much higher score. Is there any advice on how to, the timing on the best way, the best time to ask this question of your users?
3. SESean Ellis
  What I recommend is a random sample of people who've, uh, really used your product. So they, they've, they've gone in, they didn't just sign up, but they went in and, and, and hopefully, like, hit that activation moment. They've used it twice at, you know, two plus times and, uh, and, and they've, they've ideally used it, say, within the last week or two weeks so that it's, y- you know, they haven't churned yet. So if it's a random sample of those people, that's, that's kind of the ideal time to ask it.
4. LRLenny Rachitsky
  Got it. So basically it's people that have activated whatever that means to you and have been-
5. SESean Ellis
  Yeah.
6. LRLenny Rachitsky
  ... using it for a couple weeks.
7. SESean Ellis
  Yeah.
8. LRLenny Rachitsky
  Not n- not people, like, landing on your home page, not people just signing up.
9. SESean Ellis
  Right.
10. LRLenny Rachitsky
  Not people months later.
11. SESean Ellis
  Not people who've seen, like, a demo of your product.
12. LRLenny Rachitsky
  Right.
13. SESean Ellis
  But it's, it's people who actually have experienced the product. But it's okay if they've u- if, if you're hitting people who've used it months later, but like in that Lookout example that I gave, if I'm, if I'm testing people's, uh, perception of the product after I made updates to the onboarding, I'm gonna only want to, I'm gonna only want to survey people who went through the new onboarding.
14. LRLenny Rachitsky
  Yep. In the, in the experimental group?
15. SESean Ellis
  Yeah.
16. LRLenny Rachitsky
  Okay.
23:46 – 27:13
When not to use the Sean Ellis Test and other caveats
1. LRLenny Rachitsky
  So I asked people on Twitter what to ask you. A lot of people had a lot of awesome questions. I'm gonna ask, uh, I'm gonna sprinkle in a couple of these questions throughout the chat.
2. SESean Ellis
  Sure.
3. LRLenny Rachitsky
  One came in from Shreyas Doshi, a popular-
4. SESean Ellis
  Yeah.
5. LRLenny Rachitsky
  ... guest of the podcast and-
6. SESean Ellis
  One of the ones that I listened to recently. (laughs)
7. LRLenny Rachitsky
  Amazing.
8. SESean Ellis
  Yeah.
9. LRLenny Rachitsky
  Sec- I think it's the second most popular episode behind-
10. SESean Ellis
  Yeah, yeah, yeah.
11. LRLenny Rachitsky
  ... Brian Chesky. Um, okay, so he had a question of just what are the limitations of the score? When does it break down? When should you not use it, if ever? Is there anything of just like-
12. SESean Ellis
  Yeah.
13. LRLenny Rachitsky
  ... here's when it's not gonna work for you?
14. SESean Ellis
  I think one-off products would probably, you know, like how would you feel if you could no longer watch the movie you just watched? Like, I wouldn't care. Um, even like when I run a workshop, I don't run, I don't run this, this as part of my survey after I do a workshop because, um-... they're like, "How would you feel if you could no longer attend the workshop you just attended?" It doesn't ma- make sense. So I'll ask an NPS, uh, question as my, as my filtering question so that I, I'm, I'm looking at, you know, focusing in on feedback of people who love it, also then through a separate lens looking at people maybe who, who would be my detractors. So I think, I think one-off products are probably not, not good products to, to run the question on. There may be other, other, uh, places as well that I'm not thinking of right now, but that, that would be the main one.
15. LRLenny Rachitsky
  Okay. But it sounds like not many. This... What I'm hearing is it's generally widely applicable.
16. SESean Ellis
  Yeah, I think it is, like, um, at least from, from my (laughs) perspective. Like I, uh... It's, it's been really useful for me anyway.
17. LRLenny Rachitsky
  Awesome. Okay. And then the follow-up question from Shreyas is, and I kind of asked this but I'm curious if there's anything more here, if just have you seen any instances of startups over-relying on the score of prematurely declaring product-market fit when in reality they haven't reached it yet and just... Are there any other caveats of, like, "Cool, I got 40%." Is there anything else you should know? Like, okay, but maybe check this one thing.
18. SESean Ellis
  Yeah. I mean, I think to me it's kind of, like, w- what really is the definition of product-market fit? Is the definition that people who get through my crappy onboarding and actually experience the product love it, and, and if I'm able to retain those people, that means I have product-market fit? Or is fixing that crappy onboarding part of getting to product-market fit as well? I think that's up for debate. So I... You know, to me the hardest... I wouldn't obsess on onboarding if, if I know those who, who kind of get, get through the challenge of getting started with a product still don't like the product. Then it feels like it's a, a, a core product issue or, you know, wrong people using it in the wrong way issue. But, you know, once, once you have that then, then ultimately it doesn't mean that you're, you're ready to grow. Like, when, when I focus on growth then, ah, customer acquisition's almost the last step. Like, once I, once I validate that it's a must-have for those early users, then I'm thinking about, okay, how do, how do I optimize speed to value? How do I make sure that people have the right prompts to come back and use the product at the right time? So it's kind of more of that engagement loop. How do I, how, how do I get my existing users to bring in more users if, if there's something that makes sense on that end? Even how do I, how do I optimize my revenue model? And, um, once all of those things are working well, then I'll obsess on the customer acquisition side. But, um, like, customer acquisition is so hard that if you, if you're not really efficient at converting and retaining and monetizing people, you're gonna really struggle on the customer acquisition side.
19. LRLenny Rachitsky
  Yeah. Cool. And we'll talk about, uh, customer acquisition/growth.
27:13 – 36:13
Defining your own threshold and how the Sean Ellis Test came about
1. LRLenny Rachitsky
2. SESean Ellis
  Sure.
3. LRLenny Rachitsky
  Um, another question I wanted to ask and a couple listeners asked is the 40%. So I had, uh, Jag from Nubank on the podcast. I think you may have worked with them and he- they use 50% as their threshold, because, uh, apparently Brazilians are very nice and they just want Yeah. Optimistic. (laughs)
4. SESean Ellis
  Sure are optimistic.
5. LRLenny Rachitsky
  I think is what he said, yeah. Yeah. So, uh, I guess the question is do you find instances where you should increase that percentage and/ in B2B is anything different? Uh, do you change the percentage in B2B? Any advice there of just, like, when you adjust the threshold?
6. SESean Ellis
  Yeah. I hadn't really thought, thought too much on that.
7. LRLenny Rachitsky
  Mm-hmm.
8. SESean Ellis
  Um, again, for, for me, you know, generally I'm, like, uh, I'm, I'm trying to just figure out is this a product that can grow? So, like, when I'm, when I'm using it, I'm, I'm trying to just, uh, you know... So if I got a 37%, am I gonna be like, "Oh, no, this would be impossible." Um, you know, or if I g- had a, a 70%, does that mean I guarantee, like say, "Oh, yeah, I want to jump in and work with this company." Um, like, it's, it's more nuanced than that. Like, like, obviously, um, if, if it's a 70% but I have, I have no idea how I grow the business, that... I'm, I'm gonna be stuck there. So I... But I do think he, he brought up a really good point, that, that culturally some people are gonna be more, um, more optimistic or pessimistic. Interestingly, when I came up with the question, I, I used to... I used to just use kind of a normal satisfaction question. I was... When I was working at Xobni, I, I would... I'm, I'm just, like, an intensely curious person anyway, so I'm just trying to dig in and understand the customers. And so I've always done lots of surveying. But at, at Xobni, I was gonna ask it as, uh, like, use my filter as a satisfaction question. So, you know, "How satisfied are you with this?" "I'm, I'm very satisfied." "I'm somewhat satisfied." And I... Well, our, our main customers were actually senior management, and so I thought, "You know, senior management's never satisfied. I'm gonna get always this, like, super lukewarm thing. How can I, how can I change this question to be more... uh, you know, gi- give me a more kind of real answer from these guys?" Well, if I, if I flip it and say, "How would you feel if you could no longer use this product?" I'll probably get a, a more honest answer back from them. And, and of course they're very disappointed if they can't get what they want. And so initially it was just for the, for the case of, of Xobni, but then I went to, to Dropbox right after Xobni and, and like, "Oh, I'll, I'll try the question again," and the insights I got back were really useful. And so each company I went to, I g- I kept using the question. I'm like, "This works way better than your typical satisfaction question." But, but initially it was, was more about thinking just, you know, senior management to, to get a more honest answer out of them.
9. LRLenny Rachitsky
  So that's the origin story right there.
10. SESean Ellis
  Yeah. (laughs)
11. LRLenny Rachitsky
  Wow. That senior managers are just very harsh and they, th- they're n- they don't need anything.
12. SESean Ellis
  Yeah.
13. LRLenny Rachitsky
  And you have to flip it. That is so interesting. Just, like, that question is such a good reminder of how hard it is to build anything people really would be disappointed not to have. Like, that's why this works so well. People are like, "Ah, I don't need this a- app. Who cares?" Right? Like, that's the core of this, is just that is hard.
14. SESean Ellis
  I mean, especially when I first moved to Silicon Valley. So the first, like, 15 years of my career were not in Silicon Valley, and so that was... I was in Eastern Europe and then New York and then Boston. You move to Silicon Valley and you have, you have people who get really excited about technology for technology's sake, and so...You know, just something being cool is, like, isn't it cool that we can actually do this? You know, drives a lot of people. And so, you know, to, to me, I'm, I'm, I'm very, like, practical. If, if, if it's not something that, that is really bringing value to people, then, then the likelihood that that product's successful long-term is gonna be pretty low. And so... Even interestingly at, at Dropbox, I... Through the six months I was there, I would ask, I'd ask one question, like, like, multiple times a month, um, to... I, I broke kind of the, the early beta users into, into a bunch of different lists, and I'd ask, "Which best describes you? I like to be among the first to try cool new technology or I only try things that I think will be useful for me?" And over the six months, it flipped from 90% being people who try things, that they wanna try cool new technology to, uh, six months later, uh, it was people who only, you know, only are gonna try something that they feel like is, is useful. But what's kind of cool is just because what motivates you to try something is, like, you're an early adopter and you wanna try something cool, if you're gonna keep using it, it's because it's giving you some utility. And so I could, I can still use those early adopters to, uh, help me figure out where's the value inside the product.
15. LRLenny Rachitsky
  Mm-hmm. Awesome. So actually two questions along those lines. How durable do you find this percentage being? Say you hit 40%, how often does that fade and go away versus stay there or go higher?
16. SESean Ellis
  Yeah, I haven't seen it really, like, fade back down, but it's, um, but I've seen companies fail, w- you know, despite having it. And I think a lot of times then it's, it's, uh, you know, it's... It becomes like an execution challenge once you, once you have product market fit. You know, not everyone's gonna be a, a good executer. But before that, like, I, I think getting the product market fit, uh, obviously there's, there's a lot of methodology for, for, for doing it today that, that might make it a bit easier for people, but I still think it's fairly random and (laughs) pretty dang hard. And so ultimately, like, the risk factor of creating something that people care about is, is really difficult. So if you can get to the point where you have 40% of the people who are using it saying they'd be very disappointed and, and you have a reasonable sample size, let's say, you know, you got 10 people and four of them said they'd be very disappointed without it, you're still gonna get something useful from those four, but I wo- I wouldn't say that's a sample size that you can, you can really, like, go to market on. So yeah.
17. LRLenny Rachitsky
  What's a good sample size you look for, just like, "We gotta... This is actually good data I think might want to rely on"?
18. SESean Ellis
  It's really funny, like, I... And so, so so much of this stuff I kind of, like, self-learned, but I, I, I basically at one point said, "I need at least 30 responses." And, you know, I just thought I'd just kind of randomly made up a number and then, and then I had people telling me, "Yeah, 30 is kind of the minimum that you want on stuff." Like, "Okay."
19. LRLenny Rachitsky
  (laughs)
20. SESean Ellis
  And even when I first created this survey, I remember showing it to the, to the, uh, co-founder of SlideShare, and she was like, her PhD was in, you know, survey-related stuff like cognitive psychology, but she basically said it was, it was really about surveying. And she's like, "This methodology is amazing. How'd you come up with this?" And so having some of that validation around these things helped, but, you know, a lot of it was just, again, driven by my own curiosity and, um, and also just knowing that the failure is such a likely outcome that, you know, trying to, trying to reverse engineer that failure and the, and the number one reason for failure would be that people don't actually care about the product. And so when I find that, that's, that's a, that's a really good sign that it's... we're, we're now down to an execution challenge.
21. LRLenny Rachitsky
  And there's this obvious element of you may have product market fit with people, but that group might end up being very small, and the business you build around it could actually be cool but it's not gonna be a massive business. Is there anything there you can share, just like it's hard to know the size of opportunity even though you know some people who really, really like it?
22. SESean Ellis
  Yeah. I talked about I, I, I go to a, uh, multiple choice after I initially use, um, open-ended to... open-ended questions to sort of crowdsource the different use cases, but then I try to force people in a bucket, and then I can run filters on each of those buckets, and they'll be like, "Oh, people who use it this way are, like, 60% likely to be very disappointed without the product, but people who use it this way are 35% likely to be very disappointed, but way more people use it the 35% way." And so then, then having to, like, you know, do you, do you want that, like, intensely loyal group or the, or the much broader, much broader group that's, that's maybe a bit less but almost there? And I think that becomes a bit of a strategic, um, a strategic conversation of, of, like, you know, do we, do we wanna have a better chance of surviving by going after a, a niche that, that we know we can serve well, or have we, have we raised so much money that we have to go after, like, a really big market and, and that that's not... that one's not gonna be long-term? But maybe, maybe then you're, you're like, "Okay, once I, once I have traction in that market, I can, I can start to try to appeal to some other markets." So... But I think that's where kind of some strategic decisions come in.
23. LRLenny Rachitsky
  Do you have any heuristic of which you often recommend, or is it very dependent on the situation?
24. SESean Ellis
  I prefer kind of a, a, a more passionate customer base that's... and, and, and work from there, just because I think your biggest competition when you're, when you're really innovating is, is just, like, being irrelevant and... (laughs) And so if you're, if you're, like, deeply relevant to anyone, I think that gives you a much better chance of long-term success.
25. LRLenny Rachitsky
  Awesome. That's a really good
36:13 – 37:30
Tools for implementing the survey
1. LRLenny Rachitsky
  insight. Okay, two more questions along this line and then I wanna talk about growth strategy. Uh, one very tactical question. Is there, is there a tool you recommend for doing this sort of survey? Like, do you recommend inline in the product, an email, something else?
2. SESean Ellis
  I've used a lot of different, uh, tools. I actually, uh, had a survey business that I sold to private equity years ago and, and, um, that was a on, on-product, uh, you know, in... It was... It's called Qualaroo that's, like, kinda, uh-... inflow, uh, survey tool. I don't th- you know, I, I, I think just, like, s- using SurveyMonkey with, uh, with emailed surveys works fine, and for me it's a lot more of, like, uh, w- what do I... What, what's, like, pleasant for the customer to fill out, and then how, uh, w- what's gonna give me something where I can, I can work really easily with the data? So at Bounce, for example, um, they, they had already Intercom in place that had just introduced surveying, but it was kind of a really crappy customer experience on, on... at least at that time. Uh, tha- that's, that's been almost a year now, uh, or, or actually a little over a year. And, um, so I'm, I'm really sensitive to, like, is it a good survey experience for the consumer itself? And then, um... But yeah, I th- I, I don't think I'm, I'm stuck to any one platform there.
37:30 – 39:13
Transitioning from surveys to retention cohorts
1. SESean Ellis
2. LRLenny Rachitsky
  Such an important topic.
3. SESean Ellis
  Yeah.
4. LRLenny Rachitsky
  Just, like, again, to remind people why this is so important, one of the most common questions founders ask is, "Do I have product-market fit? Have I built something people want?" Like, that's just-
5. SESean Ellis
  Yeah.
6. LRLenny Rachitsky
  ... an endless series of, "I don't know. How do I know? When do I know?" And this is telling you in a really interesting way. So your advice is this is a leading indicator. You don't actually know until people actually start using it and whether they retain and continue using it. Is there just, like, uh, advice on the shift you make from relying on the survey to actually looking at retention cohorts? Is it just once you have enough data, once you have a couple cohorts, then start looking at that, forget about the survey?
7. SESean Ellis
  Yeah, what I would say is, um, uh, like but retention cohorts don't, don't give you any of the qualitative insights into the why, so that's why we continue to do the survey. So initially, I would say if the survey comes back and it's, you know, i- it, it shows y- whatever your target number is, if you wanna be like NewBank and be at 50% or you, you know... I, I spent a few years... li- two of the companies I launched, we launched in Hungary, and I would say it's kind of the opposite end of the spectrum of, of, uh, Brazil. Maybe, maybe more pessimistic than the, than the average kind of culture. And so maybe, maybe 30% is good enough there. But, but that ultimately whatever your target is that you, you, you have the signal that says, "Okay, we, we have enough value here. Let's start working on growing the business." But while you're working on growing the business, I would be paying attention to those retention cohorts. And if, if you're churning out all the customers who, uh, w- who were saying that they'd be very disappointed without the product, then okay, let's, let's retrench and, and rethink. Do, do we really have product-market fi- here, and what, what do we need to do to, to, to get it if we don't?
8. LRLenny Rachitsky
  Awesome.
39:13 – 40:18
Nubank’s approach
1. LRLenny Rachitsky
  Uh, and speaking of NewBank, if anyone wants to see how a company has actually operationalized this in the way they operate, that... there's an episode that we'll link to in the show notes where every new product at NewBank they build-
2. SESean Ellis
  Yeah.
3. LRLenny Rachitsky
  ... before they launch it, they wait for a 50%, uh, threshold for people to say, "50% of people would be disappointed if this product did not exist," as they're developing it, and only then do they launch it publicly.
4. SESean Ellis
  Yeah, I think they even do it down to the feature level.
5. LRLenny Rachitsky
  Wow.
6. SESean Ellis
  So if you, if you think about it, like, you know, if, "How would you feel if you could no longer use this feature?" starts to give you, again, the signal, is that, is that feature a must-have feature? And if it's not, maybe, maybe we shouldn't have it. And so, um, yeah, I, I was, I was super excited when I, when I saw how they were using the, the survey. And they, they were doing it before I engaged with them. Um-
7. LRLenny Rachitsky
  Oh, wow.
8. SESean Ellis
  But, uh-
9. LRLenny Rachitsky
  That's awesome.
10. SESean Ellis
  Yeah. They, they were doing it, I think, from pretty early on in the business.
11. LRLenny Rachitsky
  The reason they can do this is they have a lot of users. They have millions and millions of users, so they can ask some small percentage of people this question, 'cause the people hearing this might be like, "Oh my God, how many times am I gonna be asked this question when I'm using this feature?" But they have a lot of users, so it's easier.
12. SESean Ellis
  Yeah, yeah.
40:18 – 45:18
Case study: Superhuman’s strategy for increasing product-market fit
1. SESean Ellis
2. LRLenny Rachitsky
  Okay, last question, I promise, along these lines. Say someone is listening and they're like, "Okay, man, I'm getting, like, 10%. I'm getting 15%. I don't know what to do to increase my product-market fit," you should just... like a strategy of just dig into the people that are very disappointed and see what they have to say. But any other advice/what do you find often works in helping people move from, say, 10% to 40%?
3. SESean Ellis
  Yeah, so o- one of the things that's kind of cool about, um, almost like open sourcing the survey approach is, uh, is again watching, like, how, how NewBank has, uh, has evolved their usage. But, um, one of the other companies that I think, uh, used it in an interesting way is Superhuman, and they... I w- I would say that they basically ended up probably putting a lot more momentum behind the question than, than it had even before. Th- they, they posted something about how they did it on, uh, First Round Capital's blog. And what I have always said, and again it's me coming at it from a, from probably, uh, initially a marketing background, which is like I'm, I'm kind of taking the product as a, as a fixed, as a fixed thing, and how do, how do I actually figure out how to, how to market and grow this product? And product changes are gonna take a long time, and, and so kind of wh- what are the variables that I can control with a marketing background? But so one of the things I've always said is, you know, just ignore the somewhat disappoint, the people who say they'd be somewhat disappointed. They're telling you it's a nice-to-have. Like, they're, they're as good as gone, so just ignore those guys. And then but what Superhuman did was... And, and the reason that... You know, f- put one piece in the middle there before I say what Superhuman did. The reason that I say ignore those guys is that if you start paying attention to what your somewhat disappointed users are telling you, and then you start tweaking onboarding and product based on their feedback, maybe you're gonna dilute it for your must-have users, and that ultimately it becomes kind of good for everyone, but not great for anyone. And so that was my, my fear of, like, trying to, trying to read too much into the users who say they'd be somewhat disappointed. But the, the Superhuman guys actually found, I think, a good way around that where they said, "Okay, what is the benefit that my must-have users are focused on? And then of the users who say they'd be somewhat disappointed," so the, the nice-to-have users, "of those users who are also focused on that benefit-"What, what do they need in the product for it then to become a must-have for them? And so they're staying true to that kind of core benefit, but they're trying to essentially take those on-the-fence users and moving them up. And so I think their, their way of approaching that addressed what my concern was, which is, are we gonna, are we gonna kind of break it for the must-have users?
4. LRLenny Rachitsky
  That's an awesome insight. By the way, did Rahul and the team there just do this on their own or were you involved in any way in this
5. NANarrator
  No.
6. LRLenny Rachitsky
  ... super human approach?
7. SESean Ellis
  I mean, that's the same thing. Like I said, I wa- I wasn't initially involved with NewBank. I wasn't involved with them, and that, that's the benefit of... You know, we wrote about it in our book in 2017 and so I think that I got it out there, but I, I actually teamed up with the KISSmetrics team in, uh, 2000 maybe 12, and, and, and essentially published this survey on survey.io, where we just made it freely available for people and a really easy template to, to, to prepare and send out and the how-to guide on it. And it was all just, yeah, free. I think KISSmetrics was kind of using it as, uh, as maybe lead gen, and, and for me, I just wanted a, a way to, to kind of put something out for the community and, um, and so it's been out there for a long time, so it's not surprising that, um, that, that, that different companies have found different unique ways to use it.
8. LRLenny Rachitsky
  That's awesome. I think that post is one of the most popular in first round, really had an impact on a lot of people.
9. SESean Ellis
  Yeah.
10. LRLenny Rachitsky
  So just to repeat the, uh, approach you recommend for when you're digging into... I wrote this down when you were talking, for how to dig into what benefit people are finding. Your advice is it's basically a follow-up survey to the extremely disappointed people asking them, "What is the primary benefit you get?" It's a open text initially. Then once you get a collection, you do it sounds like another survey as multiple choice. Here's like-
11. SESean Ellis
  Yeah.
12. LRLenny Rachitsky
  ... five benefits.
13. SESean Ellis
  To, to a different group of people, to be clear.
14. LRLenny Rachitsky
  To a different group.
15. SESean Ellis
  Yeah.
16. LRLenny Rachitsky
  Got it. Awesome. And then it's like which of these four or five benefits is, is what you're getting out of this product? And then the question is, why is this benefit important to you?
17. SESean Ellis
  Yeah. And then you'll see, um, we have, uh, like eventually the, the survey.io got closed down, but the, essentially the template that I, I typically used was then moved to pmfsurvey.com, and so you'll see some other questions that I have on there as well, like, "What would you use instead if this product were no longer available?" And that's one of the interesting things is you start to see people who say they'd be somewhat disappointed, usually they're focused on a commodity use case and they, they know an easy alternative to switch to. So to be a must-have, it needs to be both, uh, valuable and unique.
45:18 – 48:24
Coining the term “growth hacking”
1. SESean Ellis
2. LRLenny Rachitsky
  Okay. S- Anything else on this topic of the Sean Ellis test, product market fit test before we move on to growth strategy advice?
3. SESean Ellis
  No, I think that's it. Yeah.
4. LRLenny Rachitsky
  Okay.
5. SESean Ellis
  Yeah.
6. LRLenny Rachitsky
  That was great.
7. SESean Ellis
  (laughs)
8. LRLenny Rachitsky
  I think we did almost an hour on that one topic...
9. SESean Ellis
  Wow.
10. LRLenny Rachitsky
  ... which I love.
11. SESean Ellis
  (laughs)
12. LRLenny Rachitsky
  Because I feel like this is such a powerful tool that I think people sort of know and have used, but I think there's a lot of opportunity to use it more effectively and... And all the stuff you pointed out about like it's not just get this... You have this threshold. Cool, let's move. Let's grow. It's like, this is just, this is how you figure out how to make it better and better and grow faster and faster. And it's actually a good segue talking about growth. So even though you coined the term growth hacking, you spend most of your time on the opposite essentially, which is helping companies figure out sustainable growth strategies, not just a bunch of hacks to grow for a little bit and then disappear. And from what I've seen, it's all rooted in this idea of product market fit and what helps you find product market fit, and I imagine many of the stuff we've talked about.
13. SESean Ellis
  Yeah. Just one, one quick, uh, interjection there is that when I coined growth hacking, I, I did not think of it as a bunch of, uh, one-off hacks that (laughs) ... What, what I thought of it was much more about what, what is the way to ultimately drive sustainable growth? But it's, uh, over time co- maybe more, uh, interpreted the way you described it, but just, (laughs) just to jump in and say that.
14. LRLenny Rachitsky
  That's a really good clarification. So how did you actually initially frame it, uh, when you first
15. NANarrator
  talked about it?
16. SESean Ellis
  Yeah, I just, I just said it's, it's about, you know, looking at every single thing that you're doing and, uh, scrutinizing its impact on growth in the business. And, um, particularly I think most marketers when I first moved to Silicon Valley, m- most CEOs who were asking me to, to help their companies, they were saying, "We need help with awareness building," and, you know, I'm getting introductions from top VCs, and so, so much of, of I think, uh, the way people were approaching growth was sort of like textbook marketing, you know, marketing textbook how to, how to approach it, and, and startups just don't have the luxury to do all of those things. And so you got to be really focused on, how do I acquire customers to an experience that's going to make them want to keep using this product? And so I, yeah, maybe, maybe I picked the wrong term in calling it growth hacking, but I, I think it at least opened the conversation to, to getting more people thinking about maybe, maybe we should be thinking about growth in a different way than as it's traditionally taught in marketing courses in school.
17. LRLenny Rachitsky
  Is there another term you think you should have used? Do you ever just think back, "Oh, I should have called it this"? Is there anything that you've had in your mind?
18. SESean Ellis
  I don't. I, I, I think, um, I think sometimes having something that's a little divisive is, uh, is almost better because it, um, i- it's too easy to just go completely unnoticed and, and so... Yeah, but I, I was trying to put a, a name on not just how I was approaching growth, but seeing, you know, Facebook obviously had a very different approach to growth than, than most companies. LinkedIn, Twitter, there were, there was a handful of companies that were approaching it in the same way I had previously been approaching it and I just thought, "We need... This thing needs a name." And so sat down with a couple of friends, came up with a name, and it stuck. And, um, but, but yeah, obviously from, from day one, it was pretty, pretty divisive with, (laughs) with different groups.
19. LRLenny Rachitsky
  That's a fun story. Thanks
48:24 – 57:25
How to approach growth
1. LRLenny Rachitsky
  for sharing that. Okay, so talking about growth and helping companies figure out how to grow. So say you go to a company. They're getting 40, say 2%, 42% on the Sean Ellis test and they're like, "Okay, cool. Let's start thinking about growth." What's your first piece of advice to them to start when they're s- thinking about growth? And then just broadly, how do you approach helping them figure out how to grow?
2. SESean Ellis
  Ultimately, it's, it's about trying to get as many of the right people to that same state that we just talked about with the...... with a must-have user. So trying to get as many people to experience the product in a way where they'd be very disappointed if they could no longer use the product. And so, that's not just acquisition, which is how most companies think about, you know. Initially, it was awareness, then maybe the more developed way was like, "Oh, let's at least focus on profitable acquisition." But in my experience, uh, you know, the, the, the hardest part is, is, is really sits inside the, the product team. So at what, how do you shape that first user experience so they actually use it in the right way, and it's not so difficult that they give up, and that ultimately, like, we, we understand what makes it a must-have product? And then, and then what we're trying to do is, is build a, yeah, sounds kind of theoretical here, but I, I can go into the details on, on how, but h- build a flywheel around that must-have value. So step one would be, would be understand it. Step two, for me, is, is then figure out a metric that essentially captures units of that value being delivered. And so, when I think about a North Star metric, that's, that's what I'm thinking about, is, is like some- something that reflects how many people are coming in and experiencing that, that product market fit experience, whatever that is. And it's not just me telling them, "Here's what your North Star metric should be." It's, it's that ultimately, ultimately the team needs to decide that together. And then, and then really just diagramming. What are, what are all of the different ways that we can grow that, that North Star metric? So that's, that's where you start to actually build. I, I call it like a value delivery engine, but it's, you know, what is our, what does our onboarding look like? What's that, that a-ha moment, that activation? What, what does the engagement loop look like? Is there any referral? Like, try to capture it as it is today, and then, from there, thinking about where are the biggest opportunities for improvement, so those high leverage opportunities, and then ultimately, you know, starting to run experiments against those opportunities. Generally, I, I think I touched on it a little bit earlier, but generally, the sequence that I like to do is start with activation, because that's, tha- that one's just so critical, and it's easy to get lost in between, uh, especially for an early product. The product team's so focused on the roadmap. Like, "We're two features away from not even needing, not even needing marketing anymore. This thing's gonna take off." And then, and then the marketing team's so focused on, like, bringing new people in, but tha- th- how do you get those new people to a great first experience kinda falls through the cracks a lot of times. So a lot of focus on, on activation, and then, and then engagement and referral, and getting, getting the revenue model right. And then once, once each of those pieces are, are working well, then starting to really obsess on, on the channel side. One thing that I'll say, like, when I go in and, and directly am involved with a company, on the acquisition side, I am thinking about my h- hypotheses on, on the acquisition pretty early on, because if I go into it and I have no idea how we'll acquire those customers, I'm not, I'm not real confident I'll figure it out when I'm there. So I wanna have, like, two or three things that seem pretty viable as, as ways to, uh, profitably acquire customers. And knowing that once I get deep into it, I'll probably come up with one or two more, and if I've got, like, five, one of them's likely to work. But I, I don't wanna, like, just be under the pressure of having to come up with that once I come in if I, if I don't at least see an angle from, uh, from, from, uh, before I get involved with a company.
3. LRLenny Rachitsky
  What I'm hearing is when you come into a company and they're asking, "Sean, how do we, how do we figure out how to grow this thing?" You actually focus first on activation, onboarding, and we're gonna talk about all these things. Then, after that, like basically these are priority order for you.
4. SESean Ellis
  Yeah.
5. LRLenny Rachitsky
  Then it's flywheel engagement, referral stuff to see if there's-
6. SESean Ellis
  Mm-hmm.
7. LRLenny Rachitsky
  ... a way to drive that. Then revenue. How do we make money with this, and how do we ma- make sure we're doing some of this profitably? And then, only then does- do you start to go big on acquisition, top-of-funnel growth.
8. SESean Ellis
  Yeah. I may need to do some acquisition stuff before just to bring enough flow through, but I'm not, I'm not obsessing on, like, how scalable is this. It's just like, yeah, let's get, let's get enough people coming through that we can, we can start to, uh, take, take the slack out. Um, part of it comes down to that the acquisition side is so competitive now, that if you're not really efficient at converting and retaining and monetizing customers, you just, you can't find scalable, profitable customer acquisition channels.
9. LRLenny Rachitsky
  This is fascinating because I think a lot of people are- probably do the opposite. Start driving a bunch of growth through our product, then we'll fix onboarding, then we'll figure out how we're making money, and referrals comes along there. So I think this is really important for people to hear. So again, th- the reason you invest first and focus a lot on onboarding/getting people activated is because that is very correlated to retention, and this must-have customer that's like, "I'll be a very disappointed customer."
10. SESean Ellis
  Yeah. And that's- and that's- and they're- you're- they're at highest risked- risk of losing them at that point. They, they're probably a little skeptical about a promise that you put out there, but they're intrigued enough to wanna use it. But until, until you get them to that must-have experience, until y- till you kind of get them to that a-ha moment, they, they're, they're at high risk of, of being lost. And so a lot of people focus on, "Well, I better get their email address or their phone number." But then it- then you're essentially having to reacquire them at that point. So to, so to me, it's just that if- if you can collapse that, that time to value... Um, I, I can give you a couple of, like, i- incredible examples of, like, what, when we f- So at LogMeIn, when we initially tried to grow the business, I was stuck at being able to spend, you know- I couldn't spend more than $10,000 per month profitably, trying to grow the business, and, uh, then I dug into the data, and I, and I saw that 95% of the people signing up for LogMeIn, so, um, LogMeIn, free...... at the time, free remote access for your computer. And, uh, so you install software and you can control it from any other computer. So 95% of the people signing up, never once did a remote control session. And so not surprisingly then, um, I, I had to get my kind of monetization off the 5% who did. That was really limiting my ability to, to find channels that worked. And so our, uh, credit our CEO with this, that I, I shared the data with him and he basically told the product team, "We are putting a complete freeze on the product development roadmap." So every single person from product, engineering, design, and then also said to me, "Stop trying to find new channels. The, the three of us on the marketing side are all gonna focus on improving the signup to usage rate." And so in three months, we improved the signup to usage rate by 1,000%. So we went from only 5% of people using the product to 50%. And went back, tried the exact same channels that previously only scaled to, to $10,000 a month. Now they scaled to a million dollars a month with a three-month payback on marketing dollars invested. 80% of new users were coming in through word of mouth. So there was this, this like major inflection point by just focusing on activation.
11. LRLenny Rachitsky
  (instrumental music) This episode is brought to you by Merge. Product leaders, yes, like you, cringe when they hear the word integration. They're not fun for you to scope, build, launch, or maintain. And integrations probably aren't what led you to product work in the first place. Lucky for you, the folks at Merge are obsessed with integrations. Their single API helps SaaS companies launch over 200 product integrations in weeks, not quarters. Think of Merge like Plaid, but for everything B2B SaaS. Organizations like Ramp, Drata, and Electric use Merge to access their customer's accounting data to reconcile bill payments, file storage data to create searchable databases in their product, or HRIS data to auto-provision and de-provision access for their customer's employees. And yes, if you need AI-ready data for your SaaS product, then Merge is the fastest way to get it. So want to solve your organization's integration dilemma once and for all? Book and attend a meeting at merge.dev/lenny and receive a $50 Amazon gift card. That's merge.dev/lenny.
57:25 – 1:05:17
Improving activation and onboarding
1. LRLenny Rachitsky
  What do you find often works in helping increase activation? I know there's, there's a million things that people do and, but I guess, what, what are like three or four things that you think people should definitely try to help improve activation in their onboarding conversion?
2. SESean Ellis
  One of my favorite quotes is a quote from a guy, Kettering. It was early, like a hundred years ago at GM running innovation, and, and he says, "A problem well stated is a problem half solved." And so I think a lot of it comes down to not the things you try, but how you deeply understand the problem that's preventing someone from getting, using your product effectively. And so, yeah. I, I'll just give you one example. We, we had one channel, like k- kind of after we made a lot of these changes and had already drove, driven a ton of improvement in, uh, in the Log Me In onboarding. We found a demand generation channel that was like really cheap and that the economics looked great. But at the, just the download step, we had a 90% drop-off rate. And so we A/B tested a bunch of different things there to try to, to try to improve that conversion rate. And then finally, um, we, we, you know, 10 plus tests, not, not able to improve it. Finally someone said, "Well, these people are registering. Why don't we just ask them why they signed up and didn't download the software?" And so we didn't want to do it in too kind of a creepy way, so we just said, you know, m- made it look like a note coming from customer service. We had this, this channel was sending 200,000 people a day, so 20,000, 20,000 people were converting to registering. So we had, um, essentially, uh, 20,000 people we could email, eh, and then, uh, 18,000 of them who didn't download. And so we just asked, "Hey, noticed you haven't had a chance to use the product yet." It, it looked like it was coming from customer support. "What happened?" And the answer we got back, and not a formal survey was, "Oh, I just, I just this seemed too good to be true. I didn't believe this was free." I mentioned to you we were one of the first, uh, freemium SaaS products out there. And so people were skeptical when, eh, especially in a demand gen channel where they hadn't sort of seen a radio or a TV advertisement from our competitor who was a premium only product. These were people who were kind of discovering the category for the first time. They were getting there. And so our next test, once we, once we articulated what the problem was, our next test gave us a 300% improvement in the download rate, which was a, we gave them a choice. Download a trial of the paid version or download the free version. Put a big graphical check mark next to the free version. But when they saw we had a business model and a trial of a paid version, the, the free version was credible. And so that, that essentially made that channel work for us. So I think, again, it's that combination of qualitative research, looking at how others did it. We, we had this, um, theory. Our previous company had been a game company that didn't require a download. So initially we had this theory that maybe, maybe just downloadable software can't be in the millions of, of new customers a month. And so we're, we're being unrealistic here. But then we were like, are there any counterexamples to that? And like, no, the instant messengers are downloadable and they, they have hundreds of millions of, of customers. So let's study their download and install process and see if we have any ideas that we could borrow from that. So again, some inspiration. Tried some of those things. But it was, it was a combination of just trying out a, a bunch of different stuff that, that ultimately, uh, led to ... There, there was n- I wouldn't say there was one big gain. It was, it was a bunch of small gains.
3. LRLenny Rachitsky
  Awesome. Okay. So (clears throat) a few things for people to try if they're like, "Hey, how do I improve my activation rate? How do I improve my conversion rate?" Is...Just like drill further into what is stopping people from progressing, like ask them, "Why did you bounce here? Why did, what did you think this was going to be? Why didn't you end up using this?" Look for inspiration from other products. I think people probably already know that. You talked about earlier this idea of the positioning having a big impact of just like figuring out they want antivirus software, let's make that very clear. "Hey, we're, we've got the best antivirus software. That's what we're here for." So there's probably just like messaging that you find works a lot of times, right?
4. SESean Ellis
  I mean, your two big levers on, on driving a conversion are, uh, increased desire, reduced friction. And so yeah, you, you definitely want to increase the right desire and then, uh, and so- sometimes it can also just be reminding people along the way of what benefits you're going to get. Cor- in the case of Log Me In, it was, it was probably the most complicated funnel I've ever seen because you, you couldn't even get to the aha moment while you're sitting in front of the computer. You had, you had to actually go to a different computer and to use the service to remote control the computer you're in front of. So it's not surprising that there was a, like-
5. LRLenny Rachitsky
  Hm.
6. SESean Ellis
  ... so many steps where we could lose people, but we just weren't that intentional about designing each of those steps initially. And it wasn't until we kind of thought through what, why would we lose someone at this step? And studying the data, which, which steps were we losing the most people at? Then deeply trying to contextualize why are we losing them there? Coming up with a, a, a set of tests that we want to run and then having a good way of deciding which one to, to test first and, and ultimately, uh, you know, ultimately focusing the tests on the areas where we're losing the most people.
7. LRLenny Rachitsky
  The other element of this is coming up with an activation metric and mi- landing on like here's what we consider someone activated. I know this is very dependent on the product, but any advice or heuristic for how to help people decide this is our activated user?
8. SESean Ellis
  I tend to start qualitatively, so just like when do I think they've had a good enough experience with the product to, to, to really like, like know it? And so like in the case of Log Me In, it was pretty easy. If they didn't do a remote control session, they, they didn't use the prod- there was no value along the way there. And so... And then at least try to see if there's a correlation to long-term retention of doing that. Causation is y- you need to do some experimentation to, to, to prove causation. At the very least, I want to see that correlation. But if, if I start with, you know, two or three ideas of what it might be and then, and then go and, and study the data, that, that, that can help you focus. But again, I don't think there's like necessarily one exact right answer of what is, what is that aha moment. There, there might be two or three different things. I think it's that intentionality about picking something that's experience-based and saying, "What is a likely experience that someone's going to get a, a, a, a good enough taste of this product?" I, and then I, I do see some companies that are like, "Well, the, the activation moment should be they've used it 100 times." (laughs) Like, yeah, they're h- y- th- that's going to correlate to long-term retention, but y- it's just not very actionable. It's so far down the user experience. So ideally, if there's a way that I could get them there in, in the, the first session, in the first day, that's great. And so it's sort of something that's value that can be experienced super early. But let me actually give you an example from the first company I worked on was a game company where I actually flipped it and basically instead of making a, a traditional funnel where they could play our games after they signed up, I made our games the advertisements. So basically, we syndicated our games to 40,000 websites. They started gameplay experience on the other website. Then they would get a message that they now have a qualifying score and, uh, if they, if they register, they, they'll be in the drawing for the weekly cash prize. And then, and then we could pull them into multiplayer games on the site. And so it was, it's kind of the, the strategy that YouTube used to grow, but it was like two years before YouTube introduced the, the approach.
9. LRLenny Rachitsky
  Yeah. It feels like you basically created Zynga-
10. SESean Ellis
  (laughs)
11. LRLenny Rachitsky
  ... uh,
1:05:17 – 1:10:28
Identifying effective growth channels
1. LRLenny Rachitsky
  is what I'm hearing there. Okay, so let's move further down the funnel.
2. SESean Ellis
  Yeah.
3. LRLenny Rachitsky
  So we've talked about activation, onboarding. The next phase that you focus on is basically some people call this growth loops, growth engines, flywheels. Basically, it's like the thing that helps your business grow. And something, I- I'm curious if this resonates, I found there's basically four ways to grow and usually one of these engines is responsible for almost all of your growth. So what I've seen is basically it's you're going to go through sales, you're going to go through SEO, you're going to go through virality, word of mouth, or paid growth. Does that resonate? Does that feel right?
4. SESean Ellis
  Yeah, and I wouldn't say it's necessarily one or the other. Like I, I think, um, I think, uh, Bounce is a really interesting example where SEO was super important for, for Bounce and, er, is super important for Bounce. Um, so people who are essentially saying, "Luggage storage Paris. Luggage store..." You know, all, most people when they're trying to find a place to store their luggage, they're, they're starting with Google, but at the same time, a huge percentage of the people who, who use Bounce are dragging their, their bag down a, down a street over, you know, over cobblestones in Paris and then they, then they, they pass a sign that says store your bag here for $5 a day and, and it's like, "Oh, no-brainer." And s- and so 10,000 partners around the world means that there's, there's a lot of people kind of in the right situation on the demand gen side. So yeah, one, one would be... I actually think of kind of like, um, I'm not sure how it would map to this, but like demand generation versus demand harvesting.
5. LRLenny Rachitsky
  Mm-hmm.
6. SESean Ellis
  And so, you know, one of those examples would be a demand, uh, generation example, like when you see the signs when you're passing, it's like high context, right, right place. And then obviously the demand harvesting would be anyone who's, who's Googling. And so that, there's paid, they do paid search and, and organic search there.
7. LRLenny Rachitsky
  Interesting. Uh, I don't see that, like, that sign approach work often, but I definitely have seen it work. Like Yelp I think grew in a lot of ways of just, like, little Yelp stickers on all the restaurants.
8. SESean Ellis
  Mm-hmm.
9. LRLenny Rachitsky
  DoorDash I think probably grows through that. Um-
10. SESean Ellis
  Yeah, I think every, every business could be a little bit different, but, um-
11. LRLenny Rachitsky
  Yeah.
12. SESean Ellis
  ... for Bounce it makes sense that that would be a really good, uh-
13. LRLenny Rachitsky
  Absolutely.
14. SESean Ellis
  ... growth opportunity for them.
15. LRLenny Rachitsky
  How do you help a business figure out which area to bet on? Like, whether they should go paid, whether they should go SEO, whether they should hire sales? That's, sale's probably an easier one of, like, probably B2B, you're probably gonna have to be a sales team. I guess, just to help them pick, like, here's where y- you have a big opportunity.
16. SESean Ellis
  Again, it kinda comes down, like, as I'm, as I'm going into it, I'm, I'm thinking, "What, what are the realistic customer acquisition angles for this business?" And I want to have, you know, ideally two or three that I'm, I'm coming into it with, but it's gonna... You know, obviously, like, like Dropbox is a classic one of, like, oh man, this, this product user get user is gonna be just like a, a, a classic. There, there's, you know, file share built into it, folder collaboration. There's, there's so many pieces of it that, that cross from one user to the next. But, but interestingly, it was fairly similar to Log Me In in some senses as kind of solving... Two, two businesses that are solving similar problems in, in, in different ways, where Log Me In we grew almost entirely off of paid search. And, and part of it, again, is that, uh, for, for us we had, we had a, um, competitor that was spending, you know, tens of millions of dollars a month creating the category with a premium-only product, um, through radio and TV advertising, go to my PC. They were just, they were creating all this latent demand. And so it just made sense for us to disrupt them with a freemium service and, and to i- insert ourselves in, in the flow of someone, like, "What was that thing that I heard about on the TV commercial?" And now they go and they Google it. And, you know, same thing but free, so we, we weren't really pushing for differentiation, but just really, you know, trying to harvest that. So, the... I couldn't do that at Dropbox. Like, no one was looking for Dropbox when I went there. And so, you know, we, we, we tried a little bit with, with search to see can we, can we make it work on, you know, cloud storage or backup or kind of going to some of these, like, traditional category... No, uh, w- cloud storage wasn't even a traditional category (laughs) at that point, but backup was. And it was just like the, it was fairly expensive, um, to, to be on, and there was just not that much demand there that way, and so it just made more sense to, to focus on the user get user loops at, at Dropbox. So I think, I think that basically for, for each business it's, um, it's just thinking about what's, what's unique for that, that business that, um, is, is gonna open up channel opportunities. And everyone's gonna be a little bit, I think, jaded from whatever the last thing that worked really well, they're gonna think they can, (laughs) they can apply it in their next business.
17. LRLenny Rachitsky
  Mm-hmm.
18. SESean Ellis
  But, um, af- after enough times myself I've just, you know, you, I tend to get the most inspiration by just talking to, to customers and, and finding out how, how did they find it, how do they typically find something like that, and, and that starts to give me some ideas as well.
1:10:28 – 1:12:43
The power of customer conversations
1. SESean Ellis
2. LRLenny Rachitsky
  I think that last point is really powerful, and I'm just writing it down. You said, so essentially one of your tactics is talking to users, asking them, "How did you find this product and how do you normally find products like this?" Was that the second question? I, I think that's, it's like such a, it's, like, similar to your Scharnhorst test. Uh, it's such a simple question, but it's so powerful because how else will people find your product? It's, they go to a place to find stuff like this.
3. SESean Ellis
  Yeah.
4. LRLenny Rachitsky
  And, like, they, I Google sear- I search Google for folder sharing. Like, like there's so much there that I think-
5. SESean Ellis
  Yeah. And then I think-
6. LRLenny Rachitsky
  ... it's easy to skip over.
7. SESean Ellis
  I think the reason that you don't actually kind of hear people taking the obvious route there a lot of times-
8. LRLenny Rachitsky
  Mm-hmm.
9. SESean Ellis
  ... is because, um, and I, I w- I used to be in the same thing that people tend to be either overindexed on qualitative or overindexed on, on quantitative. So it's kind of like, you know, analytics, I'm gonna get all my answers from, from testing and analytics or I'm gonna get all my answers from traditional customer research. And I, I was very much in that initial camp for the first five years of my career. I'm just, you know, measure everything and test the heck out of things and, and find stuff that works. But I had a, a VC who, um, was our lead VC at, at Log Me In who just said, "When was the last time you talked to a customer?" Just, like, pushed me to survey and talk to customers all the time. And, and, um, at first I was like, yeah, ga- gave the smart-ass answer, "I don't care what they say, I care what they do." And that's, that's what, you know, he's like, "No, you gotta talk to him." So. Then, then just to appease him, I would, I would try to have a conversation every day 'cause he was in our office a lot, and so I could say, "Hey, yeah, I talked to a customer today" when he, when he would ask me. And then, but I started finding that my experiments were so much better the more I talked to customers. And, and j- you know, eventually I, I became very much, like, the blend of, of qualitative and quantitative research leads to much better tests.
10. LRLenny Rachitsky
  That is another amazing story and insight. It's so interesting that people sometimes think of you as a growth hacker guy, experiments, data, when most of the advice you've been sharing so far is very qualitative driven, very survey driven, talking to customers driven.
11. SESean Ellis
  Yeah. And it's, it's just really hard to run good experiments when you can't deeply contextualize what's going on.
12. LRLenny Rachitsky
  I love this.
1:12:43 – 1:14:47
Developing the Dropbox referral program
1. LRLenny Rachitsky
  By the way, I, I don't know if I knew this. So you helped develop the Dropbox referral program?
2. SESean Ellis
  I was there at the time. I basically, when I, um, even when I first started talking with, with Drew before I came in, I was like, "I think th- the way we're gonna grow this business is by leveraging the really passionate customer base and that's what we need to double down on." And, uh, and we, we had tried a, a similar kind of referral program at Xobni and my friend who actually started, uh, who started Ring, uh, Jamie Siminoff had previously, uh-... had a company called Phone Tag, like (laughs) way, way before Ring. And he had actually done a lot of the testing on, you know, kind of double-sided referral programs, and, and the, the having, having incentive on both sides. And he s- he found that that worked the best. And so I, between what we had tested at Xobni and those conversations with him, I, I hadn't actually seen PayPal yet at that point, what they were doing. But yeah, that, that was kind of like, yeah, we, it, it seems like a referral program where we have incentives on both sides is the best way to go. Interestingly, you know, six months before I was at Dropbox, I was at Log Me In, and, um, I really thought about having incentivized referrals at Log Me In, but 80% of our new users were coming in through word of mouth. And, you know, I had 100 million devices connected in on our, on our system, and I wa- I was just so afraid of breaking this growth engine by adding an incentive that I, I, I didn't want to risk it. But at Dropbox it was so early that, that, yeah. I, I would, I would still say, like, no experiment is one person. It, it happened to be when I was there. I had some insights that I brought in, but, but ultimately the guy who built it, um, was actually an intern, an Albert Ni. And, uh, he, he ended up dropping out, I think out of MIT to, to stay with, uh, Dropbox for, for a few years after that. But, um, but yeah, we, we- he was kind of my right-hand guy to, uh, collaborating on growth day-to-day.

Episode duration: 1:44:25

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode VjJ6xcv7e8s

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

Add to Chrome

Sean’s background

The Sean Ellis test explained

The 40% rule

Case study: improving product-market fit

Understanding and leveraging customer feedback

Challenges and nuances of product-market fit

When to use the Sean Ellis Test

When not to use the Sean Ellis Test and other caveats

Defining your own threshold and how the Sean Ellis Test came about

Tools for implementing the survey

Transitioning from surveys to retention cohorts

Nubank’s approach

Case study: Superhuman’s strategy for increasing product-market fit

Coining the term “growth hacking”

How to approach growth

Improving activation and onboarding

Identifying effective growth channels

The power of customer conversations

Developing the Dropbox referral program

Get more out of YouTube videos.