a16z Podcast | The Basics of Growth 2 -- Engagement & Retention

1. 0:00 – 2:39

   From Acquisition to Engagement as Cohorts Change

   1. SCSonal Chokshi0:00

      Hi, everyone. Welcome to the a16z Podcast. I'm Sonal. Today's episode continues our series on growth. The first part covered the basics of user acquisition, and so this part covers more specifically engagement and retention, including, as always, key metrics and how to think about them. Joining us to have this conversation, we again have general partners Andrew Chen and Jeff Jordan, and we cover everything from how do network effects come in, to is there really a magic number or aha moment for a product, to who are the power users and what is the power user curve for measuring them? But first, we begin with what happens after the initial acquisition phase as different kinds of users join a product or platform over time. What does that mean for engagement, and how do you analyze them using cohort analyses?

   2. ACAndrew Chen0:43

      One of the things that you see is that people end up using these products very differently because the kinds of users that you're getting are changing over time. You know, when you look at something like rideshare, you know, all the early cohorts are basically people in urban areas. And then these days, all of rideshare is more like suburban or rural folks because, you know, you've saturated all of the center. And so what you tend to see is as you acquire your, you know, folks, you know, your core demographic out, that actually ends up showing up in the engagement. And so, you know, going back to the natural, like, kinda gravity to the whole thing, this gravity also hits the engagement side of things as well, and then ultimately the LTV because your users are typically getting, you know, kind of less valuable. It may take years to see this kinda play out, but that's kind of the natural, you know, law effects.

   3. JJJeff Jordan1:29

      And, and, and there is a progression in these, and particularly the ones that are really successful. Early on, it's all about getting users.

   4. SCSonal Chokshi1:35

      Right.

   5. JJJeff Jordan1:35

      I mean, just like users, users, users.

   6. SCSonal Chokshi1:36

      Right.

   7. JJJeff Jordan1:37

      If you're wildly successful at doing that, you run out of users or you, you know, you start running low on users, and you have to go to engagement. So Pinterest has a very high quality problem right now. Most women in America have downloaded the Pinterest app.

   8. SCSonal Chokshi1:52

      Oh, yeah. I've had it for years.

   9. JJJeff Jordan1:53

      So some growth can come through, okay, there are some number of women who've never [chuckles] heard of Pinterest somewhere in the country, but much more so they need to engage and re-engage the existing audience. I mean, w- we love engagement from an investor standpoint because it's just, you know, that, that-

   10. SCSonal Chokshi2:09

       It shows stickiness

   11. JJJeff Jordan2:10

       ... some, you can often hack your way into new users. It's really hard to hack your way into true engagement.

   12. SCSonal Chokshi2:15

       Keeping them.

   13. JJJeff Jordan2:15

       If someone's spending 20 minutes, you know, a day on your site, OfferUp, you know. Pinterest and OfferUp, the major investment thesis was, "Oh my God. Look at that engagement," you know, kinda thing. And, you know, if they can scale the user base, it's, it's a beautiful thing.

   14. SCSonal Chokshi2:29

       Right. What we mean by engagement is actually interacting with them and seeing their activity 'cause to Andrew's three points of acquisition, engagement, retention, the third piece is keeping them.

2. 2:39 – 5:21

   Cohort Analysis and the Growth Accounting Equation

   1. ACAndrew Chen2:39

      The way that we'll often analyze this is looking at cohort analyses-

   2. SCSonal Chokshi2:42

      Yes

   3. ACAndrew Chen2:43

      ... where we'll look at kind of each batch of users that's joining in each week and really start to dissect like, well, how active are they really, and to compare all these cohorts, you know, over time. You're basically putting the users that come in from a particular timeframe, let's say it's a week, right? And you're putting them into a bucket, right? And what you're doing is you wanna compare all of these different buckets against each other. And so what you typically do is you look at a bucket of a cohort of users, and you say, "Okay, well, you know, once they've signed up, the week after, how active are they? And what about the week after that, and the week after that?" And you kind of like can build out a curve. And it just turns out that these curves, once you've looked at enough of them, surprisingly, you know, human nature, they all look kind of the same.

   4. SCSonal Chokshi3:24

      I bet.

   5. ACAndrew Chen3:24

      You know? They kind of all kinda curve down and, and, and for the good ones, they start to flatten out and they plateau, and then for the really good ones, they'll actually swing back up and, you know, people will come back to the service. What you wanna do is you wanna compare the various cohorts against each other in time to see if you can spot any trends on how the usage patterns are increasing or decreasing. When you add a new layer to layer cake, you know, you might unlock a bunch of new behavior. You might unlock a bunch of new frequency that didn't exist before. Or alternatively, over long thresholds of time, people tend to become less active as you move out of your current segment.

   6. SCSonal Chokshi4:01

      The cohort graduates.

   7. ACAndrew Chen4:03

      Whether or not a specific cohort of users flattens out is really important, right? Because if you're in a world where they kind of slowly degrades, and then all of a sudden it'll actually go to zero, that means that you're naturally always kind of filling up the bucket. You know? You kind of have a leaky bucket. You're constantly filling it up.

   8. SCSonal Chokshi4:18

      You're always filling it up, right.

   9. ACAndrew Chen4:19

      Right? And what happens is that gets progressively harder as you... because w- you know, if you wanna keep your overall growth rate because that means you have to double, triple, quadruple your acquisition in order to, you know, counteract for that. One growth accounting equation that's, you know, often thrown around is that your net MAUs, so your net monthly active users, equals all the new people that you're acquiring, right? Minus all the people that are churned, right? And then plus all the people that you're resurrecting, you know, in-

   10. SCSonal Chokshi4:49

       Re-engaging

   11. ACAndrew Chen4:49

       ... re-engaging, exactly.

   12. SCSonal Chokshi4:50

       Right.

   13. ACAndrew Chen4:50

       That are coming back after they've churned. And so what happens is for a new startup, you are completely focused on new users because you don't really have that many users to churn. And over time, as you get bigger and bigger and bigger, what you find is that your, your churn rate starts to... You know, it's a percentage of your actives. And so, you know, the evolution of most of these companies as they're getting bigger tends to start with acquisition, then focus much more on, you know, churn and retention, and then ultimately also to layer in resurrection as well.

3. 5:21 – 7:40

   Using Cohorts to Prove Network Effects and Segmenting Them

   1. JJJeff Jordan5:21

      And the cohort curves have s- a couple other features that I love. But usually in marketplace businesses, the best models are built off of the cohort curves.

   2. SCSonal Chokshi5:30

      Oh, interesting.

   3. JJJeff Jordan5:31

      Because you have to understand that degradation and where it goes. Using cohorts really give you a sense of are there network effects? A network effect is the business gets more valuable the more users that use it. If it gets more valuable, your newer cohorts should behave better than your early cohorts-

   4. SCSonal Chokshi5:46

      Why is that?

   5. JJJeff Jordan5:47

      ... because the service is more valuable-

   6. SCSonal Chokshi5:49

      Oh, interesting

   7. JJJeff Jordan5:49

      ... given how many there are. So in OpenTable-

   8. SCSonal Chokshi5:51

      So that's kind of a tip. I get that

   9. JJJeff Jordan5:52

      ... if there's 10 times more restaurants, you're going to get a whole lot more reservations per diner because you are serving more of the need. So the OpenTable cohorts would climb up and get more attractive over time versus, you know, we talked about typically they tend to degrade over time. If you've reversed the polarity and they're growing over time, it means you've made the business more valuable, and then you start projecting forward. Okay, where could they take this?

   10. SCSonal Chokshi6:16

       What a better way to know the business is actually more valuable-

   11. JJJeff Jordan6:18

       Yeah

   12. SCSonal Chokshi6:18

       ... than thinking it's valuable and believing your own myth.

   13. JJJeff Jordan6:20

       So i- in a network effects business, we always ask, "Show us the cohorts." Everyone asserts, "I'm a network effect. I'm a network effect."

   14. SCSonal Chokshi6:27

       Yeah.

   15. JJJeff Jordan6:27

       But when you say, "Show me the data," cohort curves are, are-

   16. SCSonal Chokshi6:30

       It's like, show me the money-

   17. JJJeff Jordan6:31

       They don't lie

   18. SCSonal Chokshi6:31

       ... is now show us the cohorts.

   19. JJJeff Jordan6:32

       [laughs]

   20. SCSonal Chokshi6:32

       I get it.

   21. JJJeff Jordan6:33

       They don't lie.

   22. ACAndrew Chen6:34

       The other really interesting part is segmenting it.

   23. SCSonal Chokshi6:36

       I was about to actually ask you what are the buckets of cohorts? Are they all demographic data?

   24. ACAndrew Chen6:40

       For a bunch of hyper-local type businesses, the reason why segmenting it based on market geography, why that's so valuable, is because then you can compare markets against each other. You can say, "Well, you know, this market, which has much more density in terms of the number of scooters, behaves like this." And you can start to draw conclusions, you know, sort of a natural AB test in order to do that. And I think the similar kind of analysis you can do for B2B companies is for products that have different size teams using it. If you have a really large team that are all using a product, well, are they all using the product more as a result? And let's compare that to something that-

   25. SCSonal Chokshi7:15

       Interesting

   26. ACAndrew Chen7:15

       ... maybe only has a couple, right? And so this way you can start to kind of disassemble, you know, the structure of these networks, and do they actually lead to higher engagement?

   27. JJJeff Jordan7:22

       Slack would be a perfect example of that. You know, just if you have t- five people in the organization using Slack, you get one use curve. If you have... You know, if the organization, it, it's the operating system for the organization, you have a very different curve.

   28. SCSonal Chokshi7:34

       Though it's not just an accident. You have to sort of architect it, not just expect, like, serendipity to fall into place.

4. 7:40 – 10:37

   The Aha Moment and the Magic Number

   1. ACAndrew Chen7:40

      So after you get the new users, the way that you have to think about it is around quality, right? You have to make sure that the new users turn into engaged users. One of the things people often talk about is just sort of this idea of, like, an aha moment or a magic moment where the user really understands the true value of the product. But often that involves a bunch of setup. So for example, for all the different social products, whether that's Twitter or Facebook or Pinterest, et cetera, you have to make sure that when you first bring a new user in, they have to follow all the right people. They have to get... We have to figure out-

   2. SCSonal Chokshi8:08

      It's like the onboarding experience.

   3. ACAndrew Chen8:10

      Right. Which, by the way, isn't just signing up, but it's actually, you know, doing all the things to get to this, like, aha, where you're like, "Oh"-

   4. SCSonal Chokshi8:17

      I get this product

   5. ACAndrew Chen8:17

      ... like, "I get this product. It's for me." And once you get that, then they're kind of... Then you have the opportunity-

   6. SCSonal Chokshi8:23

      Mm-hmm

   7. ACAndrew Chen8:23

      ... to keep them in this engaged state over time.

   8. SCSonal Chokshi8:25

      Is that really such a thing that there is, like, a aha moment, or is it sort of, like, a cumulative? A lot of the later users on Facebook came because everyone else was already there. Is this only tied to new users?

   9. ACAndrew Chen8:35

      In the case of Facebook, actually, the fact that everyone was already there makes the aha moment that much more powerful, right?

   10. SCSonal Chokshi8:41

       Mm.

   11. ACAndrew Chen8:41

       Because all your friends and family, they're already there. Your feed's already full of content. And the first time that you see photos that maybe, you know, of someone that you went to high school with, right?

   12. SCSonal Chokshi8:51

       Yeah.

   13. ACAndrew Chen8:51

       Like, that is like, "Whoa!"

   14. SCSonal Chokshi8:51

       That's actually what happened to me. I was so excited when I saw an old friend, right.

   15. ACAndrew Chen8:54

       Right.

   16. SCSonal Chokshi8:55

       Okay.

   17. ACAndrew Chen8:55

       Yeah, exactly. And so what that means is, like, you get the product, and then afterwards, you know, when you actually are getting these push notifications or emails that are like, "Hey, it's someone's birthday," or, you know, whatever, like, you've internalized what that product is. And, you know, this moment is different for all sorts of different companies.

   18. JJJeff Jordan9:11

       I've always heard this referred to as the magic number. When you show up and it's a blank slate, it's like, what is this about? But they would drive you maniacally to, you know, follow people because th- when you got to their magic number where they had statistically correlated the number of followers with long-term engagement and retention, they would kill you to get you there doing what felt like unnatural acts of, like, yeah, you log on, follow, and you say no, and say yes.

   19. SCSonal Chokshi9:38

       [laughs]

   20. JJJeff Jordan9:38

       And then, you know, and they-

   21. SCSonal Chokshi9:39

       Yeah.

   22. JJJeff Jordan9:39

       But, but when they got you there, it kicked in and, and the service then, quote-unquote, "worked for you." A lot of the entrepreneurs I work with are trying to figure out what is my magic moment that then creates the awareness of the value prop? So take the example of Pinterest. Pinterest, when it goes to a new market, first of all, they figured out they need a lot of local content to make it compelling to local users. A US corpus of images doesn't necessar- is helpful in international markets but isn't sufficient. And so they need to supplement it.

   23. SCSonal Chokshi10:04

       You're right. If I'm India and I want, like, sarees, I don't only want, like, skirts-

   24. JJJeff Jordan10:08

       Yeah, yeah

   25. SCSonal Chokshi10:08

       ... which you may not be able to wear in certain regions.

   26. JJJeff Jordan10:09

       Yeah, yeah, yeah, exactly. I haven't worn a saree in-

   27. SCSonal Chokshi10:11

       [laughs]

   28. JJJeff Jordan10:11

       ... North America in a long time.

   29. SCSonal Chokshi10:12

       [laughs]

   30. JJJeff Jordan10:13

       But then once you have the content set, then you have to get compelling information to that individual in front of them, which, you know, you don't know the individual when they walk in the door. The faster they do that, the, the more quickly, the better the business performs, engagement goes up, retention goes up, and it works. So different entrepreneurs had to figure out what is that? What experience do they want to deliver where people get it? And then how do you engineer your way into delivering it?

5. 10:37 – 15:21

   Shifting Leverage and Climbing the Ladder of Engagement

   1. SCSonal Chokshi10:37

      Okay. So the way I've kind of come up through ... acquisition, and you've gotten new users. They get the product. You even have hopefully a way to measure that and see and track it over time. Do you want to then go into trying to get different users? Do you take your existing users? One of the things that we covered very early on is that with SAS you always want to try to take existing users and upsell them because it's way more expensive to acquire a new customer in that context. I mean, of course you want to grow your customers. How does this play out in this context? Like, what happens next?

   2. JJJeff Jordan11:07

      A lot of companies it's a progression. So almost all the early activity in a company is, "Okay, how do I get the users?" As you get users, you get more and more leverage from efforts at activation and retention and engagement. So use Pinterest as an example again. A very high percentage of women in America have downloaded Pinterest. Then the leverage quickly goes into, okay, how do I keep them engaged, reactivate the one who disappears? And, you know, their acquisition efforts in the US get de-emphasized, and all the leverage is there except as they're going international, they're still in that acquisition part of the curve. And so I think the leverage changes over time based on the situation of the company. Facebook hasn't had any users in US in forever because they have them all.

   3. SCSonal Chokshi11:50

      This kind of goes back to this portfolio approach to thinking about your users.

   4. ACAndrew Chen11:53

      Yeah. Once you have an active base of users and customers, what starts to get really interesting is to really analyze what are the things that actually set that group up to be successful, really long-term, sticky users versus what are the behaviors and profiles where users aren't successful, right? Do you actually throw your data science team on it to figure out all the different weights for both behavioral as well as the, you know, demographic and sort of profile-based stuff on there. And so one of the first things that you figure out is that each one of these products actually has this, like, ladder of engagement where oftentimes new users show up to do something that's valuable but, you know, potentially infrequent, and you need to actually level them up to something that happens all the time. For example, when you first install Dropbox, the easiest thing that you can do is you can use it to just sync your home and your work computers, right? And that's great, but really the way to get those users to become really valuable is for them to start sharing folders at work with their colleagues. Because once they have that and people are dragging files in, you know, they're really starting to collaborate on things, that's like the next level of value that you can actually have on a daily basis versus this thing that kind of is in the background that's just syncing your storage.

   5. SCSonal Chokshi13:05

      So what are some of the things that people can then do to move those users up that ladder of engagement?

   6. ACAndrew Chen13:10

      Step one is really segmenting your users into this kind of engagement map. Oftentimes you'll see this visualized as a kind of state machine where you have folks that are new, you have folks that are casual. You wanna track how much they're moving up or down in each one of these steps. And then once you have that, then the question is, okay, well, great, how do you actually get them to move from one place to the other? First, there's, like, content and education. They need to know kind of like in context that they can actually do something. So for example, if you can get your users to set their home and work for a transportation product, then you can maybe, like, figure out, you know, okay, should I prompt them in the morning to try their ride based on what the ETAs are, right?

   7. SCSonal Chokshi13:47

      Like in the app there would be like some kind of notification.

   8. ACAndrew Chen13:49

      Right. Some, like, lifecycle messaging factor in there. The second is, of course, if your product has some kind of monetary component, then you can use incentives. 10 bucks off your next subscription if you do this behavior that we know for sure gets you kinda to the next step. And then the third thing is really just, like, refining the product for that particular use case. Maybe there are certain kinds of products that are transacted all the time, and so you maybe wanna, like, waive fees or give some credits or you do something in order to get people to kinda get addicted to that as a thing.

   9. JJJeff Jordan14:18

      And the really interesting thing is, uh, the frequency with which something's consumed. I mean, eBay had enormous levels of engagement early on for a commerce app in particular. People would spend hours just browsing because early on it was about collectibles and it was about people's passions. So if someone's passionate about Depression-era glass, they will spend hours if you give them that depth of content to say, "Oh my God, I just found the perfect item." OpenTable and Airbnb are both typically much more episodic. Most people don't dine at fine dining restaurants with high frequency. Our median user dine twice a year on OpenTable, and so that has completely different marketing implications and user implications. Measurement's probably even more important in the engagement retention thing 'cause we got our data scientists to understand the different consumer journeys through our product, and then we tried to develop tactics to accelerate the journeys we wanted and limit the journeys we don't want. But in order to develop your strategy, you really need to understand how your users are behaving at a really refined level.

6. 15:21 – 19:38

   Frequency Metrics: DAU/MAU, L28 and the Smile Curve

   1. SCSonal Chokshi15:21

      So what are some of the engagement metrics?

   2. ACAndrew Chen15:23

      One really important area is frequency, just how often are you using the product, regardless of the intensity and the length of the sessions and all that other stuff, you know, just literally just frequency of sessions. We might often ask for a daily active user divided by monthly active user ratio, and that gives you a sense for, like, how many days is a user active.

   3. JJJeff Jordan15:41

      DAU/MAU.

   4. SCSonal Chokshi15:42

      You recently put a post out on the DAU/MAU metric-

   5. ACAndrew Chen15:47

      Right

   6. SCSonal Chokshi15:47

      ... and when it works and when it doesn't.

   7. ACAndrew Chen15:49

      Right.

   8. SCSonal Chokshi15:49

      There's a lot of nuances around when to apply it and when not to.

   9. ACAndrew Chen15:52

      DAU/MAU was very much popularized by the fact that Facebook used it, including in their public financial statements. And it really makes sense for them because they're an advertising business, and it matters a lot that people use them a lot all the time, right?

   10. SCSonal Chokshi16:04

       Right.

   11. ACAndrew Chen16:05

       Um-

   12. SCSonal Chokshi16:05

       It's like counting impressions and being able to sell that to advertisers, right?

   13. ACAndrew Chen16:07

       Exactly. Their products have historically been 60%-plus daily actives over monthly actives, and that's very high. You know, you're using it, um, more than half the days in a month. On the flip side, what I was talking about in my essay about this is that DAU/MAU can tell you if something's really high frequency and if it's working, but a lot of times products are actually lower DAU/MAU for a very good reason, because there's sort of just a natural cadence, you know, to the product. Like, you're not gonna get somebody who is using a travel product to use it more than a couple times per year, and yet there are many valuable travel companies, obviously.

   14. SCSonal Chokshi16:41

       So you're saying don't live and die by that alone.

   15. ACAndrew Chen16:43

       Exactly. Right.

   16. SCSonal Chokshi16:43

       'Cause it really depends on-

   17. ACAndrew Chen16:44

       Right

   18. SCSonal Chokshi16:44

       ... what product you have, the type of nature of use that it has, et cetera.

   19. ACAndrew Chen16:48

       You just wanna make sure that the metric reflects whatever strategy that you're putting in place. If you think that your product is a daily use product and you're gonna monetize using, you know, a little bit of money that you're making over a long period of time but your DAU/MAU is low, is, like, sub 15%, then, like, it's probably not gonna work. And then a metric called L28, which is something else that was pioneered certainly early at Facebook, and it's a histogram. And what you wanna do is-

   20. SCSonal Chokshi17:13

       A histogram is a frequency diagram

   21. ACAndrew Chen17:14

       ... right, frequency diagram that basically says, okay, you know, show a bar showing how many users have visited once in that month. Then twice in the month, and then three times in the month, and then four times in the month, and you kind of build that all the way out to 28 days

   22. SCSonal Chokshi17:27

       'Cause there's 28 days in the month on average, right?

   23. ACAndrew Chen17:29

       And the 28 days is to remove seasonality. And then related one obviously is like L7, right? So just like last seven days. And so what you wanna see-

   24. SCSonal Chokshi17:36

       So would this be WOWs, weekly active users? [laughs] Is that really a thing, by the way? Am I just making that up?

   25. ACAndrew Chen17:40

       Right. Yeah, WOWs.

   26. SCSonal Chokshi17:41

       Okay, there.

   27. ACAndrew Chen17:41

       DAUs over WOWs, yeah.

   28. JJJeff Jordan17:41

       You just coined it.

   29. SCSonal Chokshi17:42

       I know, right. I'm like re-coining retainment. Why not?

   30. ACAndrew Chen17:45

       [laughs] Right. And so the idea with, you know, an L28 or an L7 is the idea that you should actually start to see, when you graph this out, that there's a group of people who just use it 28 days out of 28 days, right? And that there's a big group of people who use it 27 days out of 28 days, right? And that there's a big cluster. And so that's how you know that you actually have a hardcore segment, and that's really important because in all these products, you typically have a core part of the network that's driving the rest of it, whether that's power sellers or power buyers or, you know, in a social network, the creators versus the consumers.
7. 19:38 – 21:55

   Why Engagement Metrics Are Hard to Game

   1. ACAndrew Chen19:38

      And the engagement ends up being very related to acquisition, because when you look at all the different acquisition loops, whether it's paid marketing or a viral loop or whatever, all of those things are actually powered by engagement ultimately. Like, you need people to get excited about a product in order to share content off of that platform to other platforms in order to, you know, get a viral loop going. And so one of the things I was gonna also add on DAU, MAU, and L28 is that they're actually really hard to game, right? Which is fascinating.

   2. SCSonal Chokshi20:07

      Yeah, why is this?

   3. JJJeff Jordan20:08

      Whereas growth is, can be very easy to game.

   4. ACAndrew Chen20:10

      Right. Exactly.

   5. SCSonal Chokshi20:11

      Yeah, why is that? What's the difference?

   6. ACAndrew Chen20:12

      Yeah, so the, the typical approach is to say, "Well, you know, I'm gonna add in email notifications, I'm gonna do more push notification, I'm gonna do more of this and that," and then automatically, you know, these metrics ought to go up, right? The challenging thing is actually usually sending out more notifications will actually cause more of your casual users to show up, 'cause your hardcore users were already kind of showing up, you know, already. And what that does is that'll increase your monthly actives number, but actually not increase your daily actives as much. So I- I've actually seen cases where sending out more email decreases your DAU, MAU, as opposed to increasing it.

   7. SCSonal Chokshi20:45

      That's really interesting. When you think about this portfolio of metrics, it really tells you a story about people are kind of coming but not really staying.

   8. ACAndrew Chen20:52

      If you get an email or push notification every day, eventually you turn them off and then you just stop. So then you get counted as a MAU for that period of time, and then, you know, you lose them as a DAU. And acquisition is super easy to game because you can just spend money.

   9. JJJeff Jordan21:05

      Or you got a distribution hack that works. Early on in the Facebook platform, companies literally got to a million users in, you know, it felt like minutes, just because the, there were so many people on Facebook, and the ones who were early just, it got exploding user bases. There were a number of concepts whose mean number of visits was one.

   10. SCSonal Chokshi21:24

       Yeah.

   11. JJJeff Jordan21:24

       They never came back. So you get to see these incredibly seductive growth curves, but our job is essentially to be cynical and just say, "Okay, I, we need to go below that," because there are a lot of talented growth hackers who can drive growth. I looked at a number of businesses that had tens of millions of users, and no one ever came back.

   12. SCSonal Chokshi21:43

       This is why engagement is so, so key. So we've talked especially about the fact that growth and network effects are not the exact same thing, 'cause network effects by definition are that a network becomes more valuable the more users that use it.

8. 21:55 – 24:31

   Network Effects in the Data: OpenTable and Diminishing Returns

   1. SCSonal Chokshi21:55

      What happens on the engagement side with network effects? What are the things we should be thinking about in that context?

   2. JJJeff Jordan22:00

      Typically, network effects, if they're real, manifest in data. Things like the cohort curves improve over time. Usually there's a decay. With network effects, there often is a reversal where they're growing because it's more valuable. Another smile, essentially. My diligence at OpenTable was, I looked at San Francisco, which was their first market, and sales rep productivity grew over time, restaurant churn decreased over time, the number of diners per restaurant increased over time, the percentage that booked through OpenTable versus the restaurant's own website moved towards OpenTable dramatically. Every metric improved. And so, you know, that's where it both drives good engagement, but also it just improves the investigation.

   3. SCSonal Chokshi22:39

      The value overall, right.

   4. ACAndrew Chen22:40

      One of the interesting points about network effects is that we often talk about it as if it's a binary thing.

   5. SCSonal Chokshi22:46

      Right, or homogenous, like all network effects are equal when they're not.

   6. ACAndrew Chen22:49

      Exactly, right. When you look at the data, what you really figure out is network effect is actually like a curve, and it's not like a binary yes/no kind of thing. So, you know, for example, um, I, I would guess that if you plotted the number, if you took a bunch of cities, every city was a data point, and you graphed on one side the number of restaurants in the city versus the conversion rate for that city, you would quickly find that when cities have more restaurants, the conversion rate is higher, right?

   7. JJJeff Jordan23:14

      It, it actually-

   8. ACAndrew Chen23:15

      I'm just guessing

   9. JJJeff Jordan23:15

      ... you're almost perfect with one refinement. The number of restaurants you have as a percent of that market's restaurant universe.

   10. ACAndrew Chen23:21

       Okay, right, right. Of course.

   11. JJJeff Jordan23:22

       'Cause having 100 restaurants in Des Moines is different than having 100 restaurants in Manhattan.

   12. ACAndrew Chen23:25

       Makes total sense. So not only that, what you then quickly figure out is that there's some kind of a diminishing effect to th- these things often in many cases. So for example, in rideshare, if you are gonna get a car called 15 minutes versus 10 minutes, that's very meaningful, but if it's, you know, five minutes versus two minutes, your conversion rate doesn't actually go up. If you can express your network effect in this kind of a manner, then what you can start to show is, okay, yeah, we have a couple new investment markets that maybe, you know, don't have as many restaurants or don't have as many cars, but if we put money into them and invest in them and build the right products, et cetera, then you can grow. You can do this kind of same analysis whether you're talking about, you know, YouTube channels and the number of subscribers you might have. Having more videos is better. I'm sure you can show that. If you go into the workplace and you start thinking about collaboration tools, then what you ought to see is that as more people use your chat platform or your collaborative document editing platform, then the engagement on that is gonna be higher, and you should be able to show that in the data by comparing a whole bunch of different teams.

   13. SCSonal Chokshi24:31

       Okay,

9. 24:31 – 26:11

   Engagement vs. Retention

   1. SCSonal Chokshi24:31

      so we've talked about engagement and also how it applies to network effects. Are engagement and retention the same thing? I mean, in all honesty, they sound like they would be the same thing

   2. JJJeff Jordan24:40

      There's overlap, but they're different

   3. ACAndrew Chen24:41

      Yeah, there's overlap

   4. JJJeff Jordan24:42

      Yeah

   5. ACAndrew Chen24:42

      Just to give a couple example. So weather is low frequency but high retention because-

   6. SCSonal Chokshi24:47

      Oh, right

   7. ACAndrew Chen24:47

      ... like you're actually gonna need to know what the weather is-

   8. SCSonal Chokshi24:50

      Only once a day

   9. ACAndrew Chen24:50

      ... you know?

   10. JJJeff Jordan24:51

       Yeah

   11. SCSonal Chokshi24:51

       Unless you live in San Francisco, you gotta check it like 20 times a day with all the micro climates

   12. ACAndrew Chen24:53

       Right. Right. Exactly

   13. JJJeff Jordan24:54

       Or if you live down here, you have to check it twice a year

   14. SCSonal Chokshi24:56

       [laughs] That's true. It's actually the same year round

   15. ACAndrew Chen24:58

       That's actually what it showed, was actually more that generally people didn't really check it that often. However, you are highly likely to have it installed and running after 90 days, 'cause that's, it's a reference thing. Like you might need it

   16. SCSonal Chokshi25:09

       It's so important, yeah

   17. ACAndrew Chen25:10

       Right? Versus if you look at something like games or e-books or, you know, those kinds of products, like really high engagement because you're like, "All right. I'm gonna get to f- I'm gonna finish this like trashy science fiction novel that I've been reading. I'm just gonna like get to it." But then as soon as you're done, you're like, "Okay, there's no reason why I would go back and read it again"

   18. SCSonal Chokshi25:29

       So the real difference is that engagement obviously varies depending on the product, the type of thing it is, whether it's a weather or e-book, and retention is are you still using it after X amount of time? That's right

   19. JJJeff Jordan25:38

       And different companies have different cadences. If the average user's twice a year, retention is did they book annually, other businesses, or did they come daily? But the model behind retention is completely different, and the model behind engagement is completely different

   20. ACAndrew Chen25:52

       Right. The chart that I'd love to really see is one that was a bunch of different categories that's retention versus frequency versus monetization. And I think you gotta be like really good at least on one of those axes

   21. SCSonal Chokshi26:04

       So we've done sort of this taxonomy of metrics. We've talked about the acquisition metrics. We've talked about some engagement metrics, primarily frequency

10. 26:11 – 28:54

    Time Spent and the Battle for Attention

    1. JJJeff Jordan26:11

       And engagement is also time, not just how frequent someone is, but just how much time do they spend

    2. SCSonal Chokshi26:16

       Right, time spent on site, on the p- piece, writing comments, not just page views

    3. JJJeff Jordan26:20

       'Cause very, I mean, the number of businesses that have great engagement is not high, 'cause there are only so many minutes in the day. And so you're just looking for where, okay, they're just passing time and enjoying it, and they both have obvious monetization associated with that behavior

    4. SCSonal Chokshi26:35

       This is why Netflix is so fricking genius, because when they literally invented the format of binge-watching, which you could not do, which is... I love it 'cause it's a very Internet native concept. I mean, they've literally fucked up everyone else's engagement numbers

    5. ACAndrew Chen26:46

       I think that's one of the narratives on why building consumer products is much harder these days, right?

    6. SCSonal Chokshi26:51

       And do you think it's true or not?

    7. ACAndrew Chen26:52

       Well, 'cause it used to be that you were, you know, what kind of time were you competing for in the first couple years of the smartphone?

    8. SCSonal Chokshi26:57

       Right

    9. ACAndrew Chen26:57

       You were competing against literally I'm gonna stare at the back of this person's head, or I can like use some cool app that I've downloaded

    10. SCSonal Chokshi27:04

        Right

    11. ACAndrew Chen27:04

        Right? Versus these days, you actually have to take minutes away from other products

    12. SCSonal Chokshi27:08

        Yes

    13. JJJeff Jordan27:08

        And it's typically other VMs, 'cause the top apps are almost all done by Facebook, Amazon, yeah, Google. And, you know, breaking through just, Mark calls it the first page. The people who are on the first screen are just such the incumbents

    14. SCSonal Chokshi27:24

        Right

    15. JJJeff Jordan27:24

        And sure, most people have Facebook on the screen and YouTube on the screen and, yeah, Amazon on the screen. So, you know, that competition-

    16. SCSonal Chokshi27:30

        It's hard to take that down, right

    17. JJJeff Jordan27:31

        ... it, it is a big overhang in right now in consumer investing, 'cause you have to displace someone's minutes

    18. SCSonal Chokshi27:37

        Yeah. I would add one more layer to that, at least on the content side, which is I think a lot of people make a lot of category errors because they think they're competing with like-minded players, and in fact, when it comes to things like content and attention, you're competing with just about anything that grabs your attention. It's not just other media outlets. It's, it's-

    19. ACAndrew Chen27:51

        It's Tinder. It's-

    20. SCSonal Chokshi27:52

        It's Tinder.

    21. ACAndrew Chen27:53

        [laughs]

    22. SCSonal Chokshi27:53

        It's a dating app.

    23. ACAndrew Chen27:54

        Right

    24. SCSonal Chokshi27:54

        It's, uh, something else

    25. JJJeff Jordan27:55

        I'm, I'm, I'm riding on the train for an hour. I could, you know, see what my friends are doing on Facebook, watch videos on YouTube

    26. SCSonal Chokshi28:01

        Yeah, that actually changes the time blocks. Xerox PARC did a really interesting study on micro waiting moments, and they're literally the little snatches of time, like two seconds here and there, that you might be waiting in line or doing something, so you can do a lot of snack-sized things in that period, which is also another interesting thing to think about for how people engage with various things

    27. JJJeff Jordan28:16

        So, you know, it's actually funny 'cause there's some businesses that have good engagement where it's a one session that goes on for a while, YouTube or Netflix or something like that. There are others that are multiple s- small sessions at an aggregate-

    28. SCSonal Chokshi28:27

        Like a podcast, which you might not finish in one sitting

    29. JJJeff Jordan28:28

        ... because it's the micro opportunities to-

    30. SCSonal Chokshi28:30

        Right
11. 28:54 – 30:48

    Measuring Retention and Churn for Episodic Products

    1. SCSonal Chokshi28:54

       What are some of the metrics for retention? I mean, is it just should I stay or should I go? [laughs] Like, is that the retention metric?

    2. ACAndrew Chen29:00

       And I think the big thing is the concept of churn is a tricky one. In some cases, like subscription, Hulu, Netflix, and then also in the SaaS world, whether or not you're still continuing to pay or not, right? And that's really obvious. The thing that's tricky for a lot of these consumer products, especially episodic ones, and it's actually less whether they've, quote-unquote, "churned" or not. It's actually just whether or not they're active or inactive, and whether or not that's happening at a rate that you in your business strategy have decided is acceptable or not. If every Halloween, you know how there's those costume stores that, like, open all over the place?

    3. SCSonal Chokshi29:30

       [laughs]

    4. ACAndrew Chen29:31

       If every Halloween you go back and you buy a costume, but you're inactive the rest of the time, have you churned or not? Like, it's not clear. And I would argue you've not churned because you're doing exactly what they want, which is to buy a costume every, you know, Halloween

    5. SCSonal Chokshi29:43

       Well, yeah, but it seems like it makes assessing the retention of a consumer business very difficult

    6. JJJeff Jordan29:47

       You adjust the time period that you're relevant on. If the average diner dines twice a year, so-

    7. SCSonal Chokshi29:51

       Then that's your time frame

    8. JJJeff Jordan29:52

       ... you can apply that metric. Travel's a similar thing. Airbnb is, you know, for the average user, relatively infrequent. You have to tailor your look to what are they trying to do? So if you're trying to stay up with your friends and you're doing it twice a year, yeah, that doesn't work. So Facebook has got a, a whole different set of-

    9. ACAndrew Chen30:07

       One of the things that companies can often do is to measure upstream signal. So for example, Zillow, you're probably not gonna buy a house very often, right?

    10. SCSonal Chokshi30:14

        Right

    11. ACAndrew Chen30:15

        Maybe w- you know, a couple times in your life. However, what's really interesting is they can say, "Well, you know, maybe folks aren't buying houses, but at least are we top of mind? Are they checking the houses that are going on sale in their neighborhood? Are they opening up the emails? Are they, you know, doing searches?"

    12. SCSonal Chokshi30:29

        Interesting

    13. ACAndrew Chen30:29

        Right?

    14. SCSonal Chokshi30:30

        Why do you call that upstream?

    15. ACAndrew Chen30:31

        In the funnel. You're kinda going up in the funnel, and you're tracking those metrics-

    16. SCSonal Chokshi30:34

        I get it

    17. ACAndrew Chen30:34

        ... as opposed to, you know, purchases. So even, you know, for OpenTable, it's like, okay, great, well, maybe if you're not actually completing the reservations, are you at least checking the app for availability?

    18. JJJeff Jordan30:44

        Or what's new restaurants, where I wanna dine? There, there's some level of content consumption

12. 30:48 – 32:40

    Iterating with Data: Closing Takeaways

    1. SCSonal Chokshi30:48

       So throughout this entire episode, there seems to be this interesting dance between architecting and discovering. Like, you might know some things up front because you're trying to be intentional and build these things, and then there are things that you discover along the way as your product and your views and your data evolves. How do you advise people to sort of navigate that dance?

    2. JJJeff Jordan31:08

       You iterate. You develop hypotheses, you put it out there and you test the hypothesis. You know, I think my product's gonna behave this way, and then did it. Probably the most important thing is for me, marketing can be art, marketing can be science. In the consumer internet, it's more science. Some companies can effectively do TV campaigns, large media budgets, things like that. For me, the better companies typically just rip apart their metrics, understand the dynamics of their business, and then figure out ways to improve the business through that knowledge. And that knowledge could feed back into new product executions or new marketing strategies or new something. It's constant iteration, but it's informed by the data at a level that, you know, on the best companies is really, really deep.

    3. ACAndrew Chen31:50

       Ultimately, you have a set of strategies that you're trying to pursue, and you pick the metrics to validate that you're on the right track, right?

    4. SCSonal Chokshi32:00

       Yeah.

    5. ACAndrew Chen32:00

       And a lot of what we've talked about today has really been the idea that actually there's a lot of nature versus nurture kinda parts about this. You know, your product could just be low cadence but high monetization, and so you shouldn't look at, you know, DAU, MAU. And so you have to find really the right set of metrics that show that you're providing value to your customers-

    6. SCSonal Chokshi32:17

       Mm-hmm

    7. ACAndrew Chen32:17

       ... first and foremost, and then really, you know, build your team and your product roadmap and everything in order to reinforce that. Find the loops and the networks that exist within your product because those are the things that are gonna keep your engagement improving over time, even in the face of competition.

    8. JJJeff Jordan32:32

       Growth is good. Growth and engagement is really, really, really good.

    9. SCSonal Chokshi32:36

       [laughs] Okay. So that's fabulous. Well, thank you guys for joining the a16z Podcast.

Episode duration: 32:40