Albert Cheng: Why amplifying users beats forcing virality

1. 0:00 – 4:25

   Introduction to Albert Cheng

   1. ACAlbert Cheng0:00

      (instrumental music) Growth is... The job is to connect users to the value of your product. Growth sometimes gets this reputation that it's just pure metrics hacking.

   2. LRLenny Rachitsky0:08

      You worked at three of the most successful consumer subscription products in the world. What do you think is the biggest missing piece that people don't get about building a successful consumer subscription product?

   3. ACAlbert Cheng0:18

      User retention is gold for consumer subscription companies. If you don't retain your users, then a lot of the onus is on getting them to pay on day one.

   4. LRLenny Rachitsky0:26

      Noam Levinskey, he said that I need to ask you about the biggest monetization win that you found at Grammarly.

   5. ACAlbert Cheng0:31

      The lift product experience for most of the free users was that Grammarly was just a product to fix your spelling and grammar, because those were the free suggestions. What if we actually sampled a number of different paid suggestions and interspersed them to free users across their writing? All of a sudden, people were seeing Grammarly as a much more powerful tool than they were before.

   6. LRLenny Rachitsky0:50

      What's the most counterintuitive lesson you've learned about building teams?

   7. ACAlbert Cheng0:54

      I saw some of the highest performers just being people that had very high agency, had that clock speed, had that energy, but they didn't necessarily need to have deep experience on that matter. Sometimes experience could be a crutch, especially in this world where the grounds are shifting so fast with AI. A lot of your learned habits actually need to be intentionally discarded.

   8. LRLenny Rachitsky1:12

      (instrumental music) Today, my guest is Albert Cheng. Albert is known as one of the top consumer growth minds in the world. He led growth and monetization at three of the most successful and beloved consumer products in the world, Duolingo, Grammarly, and now Chess.com. Earlier in his career at YouTube, he worked on streaming and gaming features used by over 20 million people. His unique approach to growth blends marketing, data, strategy, and product management. And in our conversation, we cover a lot of ground, including his explore and exploit framework to find growth opportunities, his biggest and most interesting growth wins at Duolingo, Grammarly, and Chess.com, how he uses AI to accelerate his growth work, what he's come to realize about the power of brand and community in your growth work, his top experimentation best practices, why his goal at every company is to run 1,000 experiments a year, and so much more. A huge thank you to Eric Alabast, Noam Levinskey, and Jorge Mazal for suggesting topics for this conversation. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It helps tremendously. Also, if you become an annual subscriber of my newsletter, you get 15 incredible products for free for an entire year, including Lovable, Replit, Bolt, n8n, Linear, Superhuman, Descript, Whisperflow, Gamma, Perplexity, Warp, Granola, Magic Patterns, Raycast, ChatBRD, and Mobbin. Head on over to lennysnewsletter.com and click Product Pass. With that, I bring you Albert Cheng. My podcast guests and I love talking about craft and taste and agency and product market fit. You know what we don't love talking about? SOC 2. That's where Vanta comes in. Vanta helps companies of all sizes get compliant fast and stay that way with industry-leading AI, automation, and continuous monitoring. Whether you're a startup tackling your first SOC 2 or ISO 27001, or an enterprise managing vendor risk, Vanta's Trust Management Platform makes it quicker, easier, and more scalable. Vanta also helps you complete security questionnaires up to five times faster so that you could win bigger deals sooner. The result? According to a recent IDC study, Vanta customers slashed over $500,000 a year and are three times more productive. Establishing trust isn't optional. Vanta makes it automatic. Get $1,000 off at vanta.com/lenny. This episode is brought to you by Jira Product Discovery. The hardest part of building products isn't actually building products. It's everything else. It's proving that the work matters, managing stakeholders, trying to plan ahead. Most teams spend more time reacting than learning, chasing updates, justifying roadmaps, and constantly unblocking work to keep things moving. Jira Product Discovery puts you back in control. With Jira Product Discovery, you can capture insights and prioritize high impact ideas. It's flexible, so it adapts to the way your team works, and helps you build a roadmap that drives alignment, not questions. And because it's built on Jira, you can track ideas from strategy to delivery all in one place. Less chasing, more time to think, learn, and build the right thing. Get Jira Product Discovery for free at atlassian.com/lenny. That's atlassian.com/lenny.

2. 4:25 – 9:37

   From classical pianist to growth leader

   1. LRLenny Rachitsky4:26

      (instrumental music) Albert, thank you so much for being here, and welcome to the podcast.

   2. ACAlbert Cheng4:33

      Thanks for having me, Lenny. Excited to be here.

   3. LRLenny Rachitsky4:35

      I'm even more excited to have you here. So, as I do for every podcast conversation, I reached out to a bunch of people that you've worked with, that know you well, to find out what to ask you about and what topics to spend time on. Jorge Mazal, who is famous in my world for writing what was, for the longest time, the most popular newsletter post on my newsletter. It's actually... People have usurped it now, but it was, like, stuck there for a long time. So here, here's what he wrote. "It is a mystery to me how Albert is able to do what he does. I am actually eager to listen to this episode and learn from him."

   4. ACAlbert Cheng5:10

      That is super nice. Thank you, Jorge. I've learned so much from him. I'm the type of weird person that likes to wake up before their kids and, like, pull up a bunch of browser tabs and look at experiments, so it was perfect that Jorge brought me into the growth world at Duolingo. Learned a ton of, uh, best practices, and he's just a great guy. Thanks, Jorge.

   5. LRLenny Rachitsky5:27

      We're already getting into these tactics.

   6. ACAlbert Cheng5:29

      (laughs)

   7. LRLenny Rachitsky5:29

      I love it. Uh, let me just give a little framing on what I want to do with this conversation. What I want to try to do is to help people learn tools and mental models for finding growth opportunities for their own products, and essentially learn the growth mentality that you bring into the companies and products that you work on. What I want to start with is, to give us a little insight into how you became what you became, there's an interesting pattern I found across a bunch of recent guests, which is, many people were very good at piano-

   8. ACAlbert Cheng6:01

      (laughs)

   9. LRLenny Rachitsky6:01

      ... when they were younger, and were very serious piano players.For example, head of ChatGPT, Nick Trelli, was like almost gonna become professional jazz pianist. You were very serious in, as a piano player earlier in your career. How did you go from pianist to one of the top growth minds in the world, briefly?

   10. ACAlbert Cheng6:18

       Well, that's very flattering, um, but I appreciate it. Yeah, I, I grew up playing a lot of piano. Um, my parents were, were immigrants from Taiwan, and I was the oldest kid that they had. And so I definitely felt that strong, um, encouragement, if you will, to learn a bunch of things, take them seriously, study hard, and so I did, right? And like my parents, even though they weren't musically proficient, they had a, like, deep love for classical music. So, I was the stereotypical, like, baby that would listen to Mozart, I guess, when I was sleeping type of thing. And I still vividly remember, like, we had this upright Yamaha piano, and at the very top of the piano, we had this countdown clock from 90 minutes. Literally every single day of my childhood, um, just practiced really, really consistently. At first, like, I really was irritated by that thing, but as I grew older, I started to appreciate, like, music quite a bit more. But anyway, like, I think what really accelerated my, my interest and abilities in, uh, piano was like, I, I feel like I hit the lottery. I had perfect pitch, and so I was able to, you know, quickly understand whether I was like playing the right stuff or the wrong stuff, and just pick up music pretty, pretty rapidly.

   11. LRLenny Rachitsky7:29

       What does per- perfect pitch even mean? Is that, does that mean you know which note is-

   12. ACAlbert Cheng7:32

       Exactly.

   13. LRLenny Rachitsky7:33

       ... is playing? Okay.

   14. ACAlbert Cheng7:34

       Exactly.

   15. LRLenny Rachitsky7:34

       Wow.

   16. ACAlbert Cheng7:35

       So I could listen-

   17. LRLenny Rachitsky7:35

       Cool.

   18. ACAlbert Cheng7:35

       ... to a song and then like just a very, very clear understanding of which note I'm supposed to start with, and if I'm-

   19. LRLenny Rachitsky7:41

       Oh my God.

   20. ACAlbert Cheng7:41

       ... playing something wrong. So it's- it's- it's-

   21. LRLenny Rachitsky7:42

       Unfair.

   22. ACAlbert Cheng7:42

       ... very helpful. It's unfair.

   23. LRLenny Rachitsky7:44

       Cool.

   24. ACAlbert Cheng7:44

       Definitely. So anyway, yeah, I got, I quite, got quite good, like, as a teenager in high school, and even considered, like, studying at a music conservatory. My intrinsic motivation for music wasn't necessarily as strong at that point, and so I decided to go to engineering school, uh, instead. But that would've been an incredibly different career. And to your original point around the relationship between, like, music and growth, I didn't really reflect on this until recently. You know, I have a four-year-old, and I'm, like, starting to teach him how to bang on the keys a little bit. But a couple things stand out. I mean, one is that I think music and growth, they both rely on this just consistent repetition. Like, you're constantly making mistakes. You have this super tight feedback loop. You have to get really resilient to just making mistakes all the time, and you know that the way of learning is through those mistakes, right? So, that's kind of a thing that I learned very early. And the second thing that, uh, occurred to me is that they both have this, like, structural underpinning to them. With growth, you have a growth model, you have metrics, you have experiments, you have channels, things like that, but you also need, on a day-to-day basis, to have creativity. You gotta come up with, like, interesting solutions and hypotheses to test. And the same is true on the music side, right? You have music theory, you have scales and stuff. But to create beautiful music, you need that passion, that emotion, that flow. Um, so I think that's the beautiful combination between the two.

   25. LRLenny Rachitsky9:09

       Fun fact, my wife bought me piano/singing lessons for Father's Day recently, and I've gotten really into this stuff. So, I'm learning how to play very basic piano now, and- and learning to, uh, identify notes and hit notes with my voice.

   26. ACAlbert Cheng9:25

       Nice.

   27. LRLenny Rachitsky9:25

       What a weird new thing.

   28. ACAlbert Cheng9:25

       Could be your next act.

   29. LRLenny Rachitsky9:26

       This could be, I could go the reverse. I could become a professional piano player. Oh man, that would so fun. It's so hard though. I'm just like, my fingers are like, "How do you, how do you do four freaking keys at once?"

   30. ACAlbert Cheng9:36

       Yeah.
3. 9:37 – 15:19

   The explore and exploit framework for growth

   1. LRLenny Rachitsky9:37

      on here?" Okay, so let's get, let's get into the meat of it. I wanna talk about growth. There's a very specific framework that, as we were chatting, that I think would be really helpful for people to hear and learn from you. You called it explore and exploit. I think there's a different, bunch of different ways to think about this. Talk about this framework and how that informs the way you think about growth.

   2. ACAlbert Cheng9:56

      Yeah. I initially came up, or heard with, heard about explore and exploit through, uh, my engineering partner at, uh, Grammarly, Nirmal, and I think he actually had taken some Reforge classes. So maybe the original (laughs) inventor of it might be Brian Balfour, who I know has been on your pod. Um, but anyway, it's a great concept. The gist of it is that when you're in exploratory mode, think of it as like finding the right mountain to climb. And then when you're in exploitation mode, it's like focusing your resources on climbing that mountain effectively. And certain companies, I think the- the warning is to basically spend too much of your time on one end of the spectrum, right? If you do too much exploration, you can have your team feel a little bit too scattershot, just trying 100 different random ideas, what's the through line, what's the strategy, how do you pattern match, you know, successes across them? And if you do too much in exploitation, which is often the MO of growth teams, it can lead to this, like, saturation and stagnation, where you're just locally maximizing a thing. And even though this principle of e- explore and exploit, like, it's typically thought of as a, as a macro thing, I like to work with my teams more on the- the micro, on the insight level. So, I'll give you a concrete example. So I work at chess.com, and, um, one of our priorities is to encourage chess players to improve, to learn and improve. So one of the PMs that we have, Dylan, he works on all the learning features. The most used learning feature in our product is called game review. So you play a game of chess. After the game's over, we have this virtual coach that teaches you about your worst moves, best moves, et cetera. And his job is to, like, improve user engagement and retention. And so he is in this exploratory phase trying to figure out, like, how do I drive more of that type of activity? And what he observes is that 80% of people that review their games actually do so after a win. And that's really counterintuitive to when we initially built the feature, we thought that people would want to use it after losses or to see their mistakes, such they could like work on their mistakes. That turned out not to be the truth when it came to the human psychology and the actual data of it- of it. And so we made some changes in the product experience. When you lose a game, now as opposed to surfacing your blunders and your, like, horrible stuff that you did, we flip it on its head, and so we show you your brilliant moves, your best moves, and we have coach say something encouraging, you know, "Losing just part of learning," like, "Keep it up," that type of thing.That change alone was pretty dramatic for us. It grew game reviews by 25%, subscriptions by 20%, user retention by a lot as well. Um, so that was fantastic. But the point is that it doesn't just stop there, right? You have to take that insight, share it broadly across the company, right? Now, adjacent product managers, like the PM working on puzzles can now think about, "Okay, how do I audit these cold patterns in my product and think about making them more positive?" Right? "I can change the success rating. I could tweak some copy, change the color of some buttons." And so you now can take this, like, experiment win and expand it out 10X across your organization, and that's the kind of exploitation phase of it. So when done right, right? You can oscillate between the two until you saturate out of exploitation mode, and then you encourage the teams to, to brainstorm and get more creative again.

   3. LRLenny Rachitsky13:11

      Amazing. Okay, so there's a lot here to follow up on. One is, is the core piece of advice when you find something that works really well, find ways to build on that learning. One is, here's an insight, it can apply to other parts of the product, "Hey teams, here's something we learned unexpected, maybe this can help you." Also just keep find more, like run more experiments in the same zone, I imagine is a part of that.

   4. ACAlbert Cheng13:34

      Yeah, exactly right. I mean, in my experience, the typical win rate, and I hate to use that term for experiments, is, is often something like 30 to 50%. Like, usually you're not actually g- like, you're trying a bunch of things, a lot of hypotheses turn out not to be true. You know, consumer products are very unpredictable like that. But when you do find a thing that breaks through the noise, and it could actually be a hugely losing experiment too, those are also super valuable, right? Surfacing those across the company, like, the original PM running that experiment doesn't necessarily need to be the person that figures out what you should do for all the other parts of your product experience, but the onus is on them to clearly articulate what their hypothesis is, what they found, such that then as, like, a growth leader, I can encourage people to kind of swarm around that and try a bunch of different ideas such that the success rate is up and the impact is up. So it's just kind of oscillating back and forth between the two that is, uh, the magic bullet.

   5. LRLenny Rachitsky14:30

      I think another takeaway here slash something that I think about when I hear what you're saying is there's often a lot more wins in an area than people expect, that you can continue to find wins and growth in something for a long time.

   6. ACAlbert Cheng14:46

      Exactly right. Yes.

   7. LRLenny Rachitsky14:47

      Okay.

   8. ACAlbert Cheng14:47

      Um, at the end of the day, like, users... I think within a company sometimes you can have this siloed approach where you break apart the product experience in, you know, 50 different ways and distribute them across different teams, and you assume that users interact with each of the different features with a different mentality. But oftentimes, that's actually not necessarily the case, and so sometimes you can surface an insight that's more, you know, human psychology based that can resonate across the entire product experience. And so I think when you can find that, um, you can double down.

4. 15:19 – 16:34

   How to know when to explore vs. exploit

   1. ACAlbert Cheng15:19

   2. LRLenny Rachitsky15:19

      People hearing this might feel like, "Okay, yes, uh, find big wins and then find more." Is there something you find that helps you figure out when to explore versus when to exploit, when you've exploited too far? Just like any heuristics or, I don't know, ways of helping people guide them along this process of exploring and exploiting.

   3. ACAlbert Cheng15:41

      One, one thing that I, I try to focus on at a company of our scale of like a chess.com, right? We're running roughly 250 experiments a year, so we're not, like, the highest in the industry, but we run a, we run a decent volume, right? And so when that happens, I invest in these, like, experiment explorer tools, and we could talk about AI as well as another way to kind of uncover and pick out these nuggets of wisdom. But basically these explorer tools can allow me to look across the spectrum of experiments that are going on, try to figure out if there are patterns between the, the hypotheses and the learnings that are happening, and if I'm starting to see, like, more and more experiments that are not statistically significant, that may be a signal to me to say, "Okay, we might have ex- kind of tried to exploit a little bit too far. Like, there might not be as much juice to squeeze. Hey guys, let's like, you know, get back to the table and brainstorm and be a little bit more divergent with our thinking."

5. 16:34 – 20:42

   Using AI to accelerate growth experimentation

   1. ACAlbert Cheng16:34

   2. LRLenny Rachitsky16:34

      Well, let me follow this thread on AI and how y- you're using AI to help you figure this out. That is very cool. Talk about that.

   3. ACAlbert Cheng16:40

      I think one of the, the latest things that we've been tinkering around with is this text to, uh, SQL capability. It's actually pretty powerful. Um, we have this data request Slack channel, where for the longest time, and this is still true today, like people will toss in all sorts of just one-off questions, you know, "How many subscribers do we have in South Africa?" Or like, you know, "How long did somebody play puzzles, like, last, last month?" Or something. And these ad hoc questions, they often take a lot of, like, human time to just go in and, you know, a data analyst needs to prioritize it and find time to go run the query. And yes, you can invest in self-serve tooling to improve at this, but also I found that AI is quite good at doing that first pass answer as well. Um, and so we're working on, like, kind of training some of these Slack bots to essentially be the, the first party, uh, provider of a lot of these answers, which makes the company as a whole a lot more data informed, I guess. And I think what's also kind of interesting is that just human nature is that if you have a question that you feel like, you know, you might be a bit embarrassed to ask or you don't wanna bother someone, you just don't ask the question, right? And so by the nature of having these tools, you get actually a pretty large explosion of questions being asked, and I think you see this in ChatGPT too, right? It's like just having a thing, right, that you can converse with that you feel, uh, comfortable in makes a huge difference.

   4. LRLenny Rachitsky18:03

      Okay, this is extremely cool. So is this something you built? Basically it's a Slack bot that gives you the SQL query, or does it actually do the analysis for you?

   5. ACAlbert Cheng18:12

      No, it does analysis, yeah.

   6. LRLenny Rachitsky18:13

      Whoa, so cool. Okay. Is this gonna be something you guys are gonna release, or is this just like somebody, you guys should just build this at every company?

   7. ACAlbert Cheng18:19

      We, we should. It's a good idea.

   8. LRLenny Rachitsky18:21

      Okay, okay.

   9. ACAlbert Cheng18:21

      (laughs)

   10. LRLenny Rachitsky18:22

       Well, there's an episode where everyone in the comments is like, "Open source this!"

   11. ACAlbert Cheng18:25

       Yeah.

   12. LRLenny Rachitsky18:26

       So we'll see if that happens again.... uh, that is very cool. Are there other examples of that kind of stuff that you've done or seen?

   13. ACAlbert Cheng18:31

       I mean, an, an adjacent example is a lot of the product managers who are like, "We are tinkering around with all sorts of different prototyping tools right now," right? It's, it's like go from an idea to a representative solution. Today, right? There's a lot of humans involved in taking an idea, writing up a spec, doing a review, doing design, et cetera. Uh, I'm sure you've interviewed plenty of people that have talked about this specific problem, right? And so for us, like we've invested a bit in at least carving out the main screens of our product experience, things like our onboarding flow, our home screen, our chess board, as an example, and building like essentially AI prototypes of those using tools like, of, of v0 or like Lovable, right? And when you have those foundational pieces, you can then share them with the rest of the company and they can use that as a starting point. And then they can try to, you know, put their ideas on top of that and then they become a lot more discussable and hopefully testable relatively soon.

   14. LRLenny Rachitsky19:25

       Now, what's in your AI stack along those lines?

   15. ACAlbert Cheng19:27

       The PMs are mostly using v0. The designers-

   16. LRLenny Rachitsky19:29

       Hmm.

   17. ACAlbert Cheng19:30

       ... love Figma, so they're using Figma Make. Uh, the engineers are using a, a combination of tools right now. Um, so Cursor, Cloud Code, GitHub Copilot. Marketing teams use all sorts of tools for translation, subtitles, you know, content adaptations, et cetera. Customer support uses Intercom Fin. So there's quite a lot of tools that are kind of used across the company. I would say, though, that something that is kind of annoying to me is that we haven't yet figured out the bridging from the tinkering to the workflow quite as seamlessly as I would like, right? And so each sub-function, even though the common, I guess, wisdom now is that AI is going to strip away this, these like functional titles, it is kind of true that based on your experience, like you may gravitate to using a type of tool more. And if that tool isn't as interoperable with some of the other tools that you need to pass down the chain to actually ship it into production, at least at our scale, right? I think for smaller startups, sure, PM should just go ship it. But for us, like we are still doing some handoffs between functions. I expect that to change over time, and we are investing in some of like, you know, design system components and MCPs and stuff to make it a little bit easier. But yeah, it's a, it's an investment and it takes time to, to smooth

6. 20:42 – 24:36

   Grammarly’s biggest monetization win

   1. ACAlbert Cheng20:42

      things out.

   2. LRLenny Rachitsky20:42

      I wanna come back to this topic of how things have changed and how you work as a product person, as a growth person across the companies you've been at. But first of all, I wanna, I wanna talk about another example of finding growth wins and monetization wins. Uh, Noam L-Lovinsky, who is chief product officer at Grammarly, you worked with him for a while, while you were at Grammarly, he said that I need to ask you about the biggest monetization win that you found at Grammarly and how you discovered the opportunity.

   3. ACAlbert Cheng21:10

      I had the pleasure of working with Noam and his product team at, at Grammarly. Um, some context first for those that don't use Grammarly. So Grammarly is a AI-powered writing assistant, and so typically people will use it as a Chrome extension or a downloadable desktop client. And basically what it does is it overlays your writing with a bunch of different-

   4. LRLenny Rachitsky21:29

      I use it. I'm a big fan.

   5. ACAlbert Cheng21:29

      ... corrections. So you're a big fan?

   6. LRLenny Rachitsky21:30

      I use it and it saves my life.

   7. ACAlbert Cheng21:32

      (laughs) . Fantastic. Glad to hear that. The... Grammarly is a freemium business model, which means that over 90% of our users are on the free service and the rest of it pay for subscriptions essentially, right? And so one of the teams, they work on subscriber conversion PM there is Kyla. Um, that team's great and their job is to figure out the free to paid, uh, subscription path, right? And so one of the realizations, one is that we weren't actually tracking, um, the events that well for the types of essentially suggestions that people were getting and how often were users seeing paywalls and stuff like that. That's kind of step number one. We had to put that instrumentation in. Step number two is that, hey, we noticed actually, um, first let me explain some of the logic. So as a free user, you basically get these underlines across your writing and if you accept all of them, um, then you see the paywall and that encourages you to like, subscribe for more nuanced features. As a free user, the main things you get are spelling, grammar, they're basically correctness things. And as a paid user, you get the, like how do you improve your tone to be more empathetic? How do you improve your writing to be more clear? How can you rewrite entire sentences? That type of thing. And so the observed behavior from all that tracking and, and data was that actually a very small percentage of our free users was deciding to accept all of their suggestions. They were more kind of picking and choosing as they go. And I wonder if your, um, experience is kind of similar too.

   8. LRLenny Rachitsky22:59

      Definitely, 'cause-

   9. ACAlbert Cheng23:00

      Yeah.

   10. LRLenny Rachitsky23:00

       ... yeah, I'm always like, "Wait, wait, stop rewriting everything."

   11. ACAlbert Cheng23:03

       (laughs) .

   12. LRLenny Rachitsky23:03

       Just like, "This part is wrong. I will fix it." Yeah. I'm very much a pick and choose-

   13. ACAlbert Cheng23:07

       That's right.

   14. LRLenny Rachitsky23:07

       ... correction person.

   15. ACAlbert Cheng23:08

       And then the second thing, which is I think equally if not more interesting is that, you know, I was at this company during this generative AI transformation, which is obviously still going on, right? And quite frankly, both the company brand as well as the like lived product experience for most of the free users was that Grammarly was just a product to fix your spelling and grammar because those were the free suggestions we were showing people, right? And so we decided to flip that on its head entirely and we said, "Okay, what if we actually sampled a number of different paid suggestions and intersperse them to free users, right? Across their writing such that they were intermingled and we would provide a limited taste of what the paid offering had to provide." And on the surface, like even though it's rational, the, the, the concern is that if we give too much of this away, then will people wanna subscribe? And we found completely like that was not the case, right? All of a sudden people were seeing Grammarly as a much more powerful tool than they were before. And our upgrade rates like nearly doubled just through this change. And so I think this is interesting just modernization learning that especially if you work on a freemium product, try to have your free product be a reflection of like everything that your product can offer you. Obviously to an extent there's some costs involved with some of the paid features and things like that, but-It generally will pay its- for itself if you're able to put your best foot forward, um, and go do that. So, that really worked well for

7. 24:36 – 28:03

   Freemium vs. trial models for subscription products

   1. ACAlbert Cheng24:36

      us there.

   2. LRLenny Rachitsky24:36

      I think this is what converted me to being a paid Grammarly subscriber (laughs) . Wow, what an- what a- a genius move. So essentially it's here's, uh, here's a bunch of improvements, but you- you get like three I think-

   3. ACAlbert Cheng24:49

      Yeah.

   4. LRLenny Rachitsky24:49

      ... max. And then it's like, okay, now you gotta upgrade. Uh-

   5. ACAlbert Cheng24:51

      It's basically like a reverse free trial, but in real time, like while you're writing, as opposed to a time-based one. So we kind of adopted some, you know, patterns that are in the industry, but molded it to Grammarly's specific use case.

   6. LRLenny Rachitsky25:04

      Right. I was gonna ask. So it's not like a full trial, it's like a, a capped trial where you get a certain number of things and then you run out and then you- they get refreshed I think once a day or something like that is what I found.

   7. ACAlbert Cheng25:16

      Yeah, you got it.

   8. LRLenny Rachitsky25:17

      Yeah. Grammarly is, uh, the best slash most devious at their upsells as- I'm always just like, goddammit, you're about- you have- I'm so close to seeing an improvement. I just- could- I just have to upgrade and just like right there, it's right there where my mouse is.

   9. ACAlbert Cheng25:32

      Yeah. Well, I'm not proud of being devious, but... (laughs)

   10. LRLenny Rachitsky25:34

       Devious in-

   11. ACAlbert Cheng25:35

       But yes.

   12. LRLenny Rachitsky25:36

       ... uh, really getting me to buy the thing. Good job. What was it, Kyla? Okay. Nice job, Kyla. It's- it's very effective. I love that. And so, uh, okay. So in terms of the free trial, I don't know, is there anything there of just, there's always this question of freemium give things away and then there's pro account you get a- there's like trial versus time, some features are limited. I don't know. Do you have like a for consumer subscription products, like here's the way to go?

   13. ACAlbert Cheng26:00

       Yeah. I think first of all, why do freemium subscription in the first place is a common question that, you know, like I've joined all these companies that are freemium subscription. Like what do I like about it, I guess? Uh, um, well one, I think it ties really nicely to like mission orientation of a lot of these companies. It's often like you wanna spread the product as wide as possible because that's why the founders built the thing, right? You're- you're trying to like improve education with like Duolingo or you know, Grammarly or chess.com. Like these are meant to be widespread products with a really wide value proposition that fits globally, right? And so obviously the lowest friction to that is going to be a free product. So that alone is part of it. Another part of it is that a lot of these products primarily grow through word of mouth and especially if you can build, you know, network effects in the product, like Duolingo has a bunch of social features or with Grammarly, like they have a bit of a B2C2B play as well. So you see Grammarly being used by teams and by companies and whatnot, right? And even if users are on the free plan, they still provide quite a lot of value in making sure that Grammarly can be purchased by a coworker or by a team member or whatever, right? So I think these things are- are usually why I lean toward make sure that the core value proposition that you're providing users is free and is sort of permanently free. And then you layer on kind of a sampling or a taste of some of the premium features that are on top of it. That's usually the sweet spot that I've seen as to the trials, reverse trials type of thing. I think it largely depends. I think if you have especially a B2B feature where you may have some lock-in reverse trials can be super powerful. You just want to get people in there. You don't need to ask for their credit card because they're using your CRM or they're investing quite a lot of time in like building out, you know, material and content. And so by the time that window drops, you actually like feel, oh man, I probably should keep this and- and start paying. I think for a lot of consumer products it's a little bit harder for that to work. And so I've typically seen more just normal free trials be- be the norm.

8. 28:03 – 32:06

   What retention rates you need for subscription success

   1. LRLenny Rachitsky28:03

      Let me follow this thread of just consumer subscription products. I feel like this is the category that every indie developer dreams of building a product in because it's easy to build, cool, I'll build an app, I'll add a paywall. And then they realize this is a lot harder than I thought from a perspective of distribution and-

   2. ACAlbert Cheng28:22

      Yeah.

   3. LRLenny Rachitsky28:22

      ... CACs and growth like that. Like, is- is that the biggest missing piece that people don't get about building a successful consumer subscription product?

   4. ACAlbert Cheng28:31

      Yeah, I mean, user retention is gold for consumer subscription companies. If you don't retain your users, then a lot of the onus is on getting them to pay on like day one. That's super hard, right? Then you're dealing with totally different business models where you're paying for users, you're trying to like aggressively upsell them before they, you know, hit any sort of habitual usage patterns with your product. A lot of apps naturally do that 'cause that's how they break the mold and get their first, you know, users to- to do it. But I don't know, I've been fortunate to join companies sort of after that initial phase (laughs) . But especially like take Duolingo and chess.com. These are organic word of mouth kind of driven businesses and in- in kind of both ways, like they grew the market, right? From a much smaller market and as opposed to it being a- a very competitive space where you're kind of competing and taking market share from others and bidding for higher terms and stuff like that. So I don't know that there's something to that.

   5. LRLenny Rachitsky29:26

      So what I'm hearing here is you need to find a way to grow through word of mouth for this to have any chance of success. And also retention needs to be very high. Do you have a heuristic of what retention needs to be for you to have a chance building a successful consumer subscription business?

   6. ACAlbert Cheng29:42

      I- I think consumer companies tend to track like essentially two main types of like user retention. There's more of like the new user one, kind of D1, D7, et cetera. Um, I think when you have your D1 retention somewhere around like the 30 or 40% mark, like that's quite solid I think for- for a consumer app. If it's much lower than that, then sometimes I might question like the intent of the user or the s- the ability for that, uh, you to, I guess acquire, just mathematically acquire enough users such that you can grow a- a big enough daily active user base.

   7. LRLenny Rachitsky30:14

      That's surprisingly low.

   8. ACAlbert Cheng30:16

      Yeah.

   9. LRLenny Rachitsky30:17

      So it feels achievable in theory.

   10. ACAlbert Cheng30:18

       It's achievable. It's achievable in theory-

   11. LRLenny Rachitsky30:20

       Okay.

   12. ACAlbert Cheng30:20

       ... but there are so many options out there in the market and people are feeling a lot of like app and product bo- bloat.

   13. LRLenny Rachitsky30:25

       And so just to be clear, you're saying 20 to 30% of people come back the next day, you're-

   14. ACAlbert Cheng30:30

       Yeah.

   15. LRLenny Rachitsky30:30

       ... you know, okay for this.

   16. ACAlbert Cheng30:31

       30 to 40. 30 to 40 per-

   17. LRLenny Rachitsky30:33

       30 to 40.

   18. ACAlbert Cheng30:33

       40%.

   19. LRLenny Rachitsky30:33

       Okay. Okay, cool.

   20. ACAlbert Cheng30:33

       I think you're in an okay place.I think even more importantly, and, you know, you mentioned, Jorge, to kick this off, but, like, you know, he wrote that very, very popular article, uh, about the growth model, right, and how, like, current user retention rate was the biggest thing for them. And I think especially if you have a product that has daily frequency, like, that's the actually... the retention that matters the most is that, like, of your existing user base that has developed a habitual pattern, how sticky is your product? And it's that retention rate that really compounds and build that... builds that daily habit. So over time, especially when companies mature a little bit, you actually focus most of your energy on the existing user retention mechanics. Um, you find that that's a much, much bigger lever. One exception is that Grammarly was a different type of product in that you install it and you don't proactively open it every day. So, that was kind of interesting to me 'cause I assume that you should always just focus on existing user retention. But for a product like Grammarly, it's actually the activation, installation, aha moment that's really, really critical and will carry the user for a very, very long time.

   21. LRLenny Rachitsky31:36

       That makes sense. Like, yeah, the stats would show someone's a daily active user because they're typing things, and that's not an accurate stat for Grammarly. The other interesting trend I've noticed across successful consumer subscription products is they always start very scrappy and very cost-efficient and spend-efficient because... I think it's because it takes them a long time to find something that's working, and they're surviving on that margin of retention to growth cost, essentially.

   22. ACAlbert Cheng32:04

       Yeah. That's right.

9. 32:06 – 34:35

   The importance of resurrected users

   1. ACAlbert Cheng32:06

   2. LRLenny Rachitsky32:06

      Yeah. The... And the retention piece, that's such a good point, it's like my newsletter is very much along these lines. It's just like, how many people are joining every day? How many people are leaving? And it's, it's a difficult treadmill to be on because people, you know, they wanna save money, they wanna spend that on Netflix and things like that. So, as amazing as you are, the people are always gonna leave. So, the trick is how do you find more people coming than going?

   3. ACAlbert Cheng32:26

      Yeah, and I think, um, just to take Chess.com example, like, I think probably 80-ish percent of our daily or weekly active users, I mean, I'll check the numbers, but something like that would be like a current... a current user or an existing user, and then a new and a, like, reactivated or resurrected user, those are actually about similar size for a company of our, our scale.

   4. LRLenny Rachitsky32:47

      Mm-hmm.

   5. ACAlbert Cheng32:47

      So, even though there's a lot of attention on that new user experience, it's actually, like, pretty interesting that the components of your active user base are actually not heavily weighed in the new user set after you mature to a certain degree.

   6. LRLenny Rachitsky33:00

      Can you explain that a little bit more?

   7. ACAlbert Cheng33:01

      Yes. So, after some period of time, you kind of stack up a lot of inactive users in your product-

   8. LRLenny Rachitsky33:07

      Mm-hmm.

   9. ACAlbert Cheng33:07

      ... right? And you also stack up sporadic users, right? People that may not have a daily habit, but they will use it, you know, once or twice a week, or once or twice a month type of thing. And so eventually that math sort of adds up where you have, let's say, hundreds of millions of kind of dormant users that are coming back, and it's actually worth spending some time making sure that that kind of resurrected, for lack of a better word, experience inside the product is really excellent, and that you find novel ways to try to bring them back. Duolingo, as an example, they did a good job of using social notifications, and so if people would use, like, Contact Sync or something, you might get a push notification that one of your, like, best friends just started using Duolingo, and that might encourage you to come back and resurrect into the product. And whether you resurrected in the product, it might be the case that your proficiency of the language you were learning, like, you were learning French three years ago, but now you, like, forgot most of it, right? And so when you open the app again, it encourages you to essentially re-place yourself, like, do another placement test and put you in the right spot. And so some of these types of mechanics for a more, more mature company can, uh, lead to pretty good ROI, I guess, is what I'm trying to say.

   10. LRLenny Rachitsky34:19

       Got it. Like, essentially, so many of your... so many people have already tried in the past that to grow you need to resurrect people that have been there, and so thinking through, it's almost like a user experience for resurrected users.

   11. ACAlbert Cheng34:34

       Exactly.

10. 34:35 – 45:53

    Differences between Duolingo, Grammarly, and Chess.com

    1. ACAlbert Cheng34:35

    2. LRLenny Rachitsky34:35

       Okay. Let's zoo- zoom out a little bit. You've worked at three of the most successful consumer subscription products in the world. What is... what is the difference between how these three operate? I think there's many ways to be successful. It feels like these companies are very different. What's kind of the gist of what each of these com- how they operate?

    3. ACAlbert Cheng34:53

       Well, first of all, like, there's obviously a lot of similarities, but I'll just focus my answer on the, the differences. So, I think Duolingo, what struck me most working there is they're very particular. They have a pro- an approach of product development that is infused across, like, everyone in the company, and they tend to... they actually wrote a playbook about this. It's called The Green Machine.

    4. LRLenny Rachitsky35:12

       Mm-hmm.

    5. ACAlbert Cheng35:12

       You can look it up.

    6. LRLenny Rachitsky35:12

       I tweeted about it.

    7. ACAlbert Cheng35:13

       Yeah.

    8. LRLenny Rachitsky35:13

       That was one of my most successful tweets ever.

    9. ACAlbert Cheng35:15

       Really?

    10. LRLenny Rachitsky35:15

        I just tweeted something about Duolingo just released their playbook, and I screenshotted, like, the- the owl's button screen, like-

    11. ACAlbert Cheng35:23

        (laughs)

    12. LRLenny Rachitsky35:23

        ... page, and it was, like, 5,000 likes.

    13. ACAlbert Cheng35:26

        That's hilarious.

    14. LRLenny Rachitsky35:26

        Yeah. So, yeah. Keep going, sorry.

    15. ACAlbert Cheng35:28

        But, yeah. I mean, the ethos of the company, I mean, they, they hire a lot of intelligent, energetic people out of college basically, and they give them a lot of amazing experimentation tooling, and they care a lot about, like, the clock speed of the company, right? So, it's a lot of creativity, lot of ideation. The product experience of Duolingo actually, like, changes multiple times per day for each user, which is pretty shocking. And so I'd never worked in a place like that before, um, but it's really struck me about how consistently the company operated, and they had specs and processes for doing, like, each of those steps in their product development cycle, and they were really, really tight about it.

    16. LRLenny Rachitsky36:08

        Okay. So, that's Duolingo.

    17. ACAlbert Cheng36:09

        Yeah. That's Duolingo. Grammarly, you know, this is an interesting company because they started as a paid product oriented at students, then they expanded into more of a freemium model tailored to everyone, gradually focusing more on the professional base. And then as they accumulate a lot more professionals, they realize, hey, there's patterns, right? We're seeing that a bunch of marketing teams or a bunch of sales teams or a bunch of customer support teams or whatever, right, particular functions within particular companies were really adopting Grammarly at scale. And so-They were able to then layer on much more of a managed, kind of enterprise-y motion. And while I was there, I was focused on the consumer self-serve motion. But they weren't siloed, right? They were, they were intermixed with each other. And so a big part of my job was not just to grow, like, the self-serve revenue and self-serve active users, but it was also, how do you uncover kind of the right, uh, teams, the right functions, the right companies for, like, demand gen and sales to go reach out to? Um, so that was a very interesting... It's kind of product-led sales work, right? It's really fascinating thing for me to learn. And then on top of that, with all the transformation going on with generative AI, and even recently with them acquiring, uh, Coda and Superhuman and becoming more of a productivity suite, like, the company is just evolving pretty rapidly. It's a really exciting, uh, thing for me to be a part of and to see from the sidelines. But that just made it, at its core, kind of a different growth job than, um, than Duolingo, for sure.

    18. LRLenny Rachitsky37:40

        Essentially a B2B business, versus a very consumer business.

    19. ACAlbert Cheng37:43

        Yeah, and a lot more meaningful strategic decisions as well.

    20. LRLenny Rachitsky37:46

        Mm-hmm.

    21. ACAlbert Cheng37:47

        And then the core product team also. You know, I'm used to, in growth, like, laying out the entire user journey that a user will go through. You know, acquisition, activation, engagement, so on and so forth, right? And typically, growth teams, if they're well-resourced, they can do enough to move each one of these various levers, right? It's just a matter of, like, the sequencing of them and what you want to prioritize first. But Grammarly was kind of unique, in that the core product experience itself was what drove repeated activity, right? It's that, I, I previously mentioned that current user retention thing. What most drives that is the frequency and the quality of the suggestions that you get every day, right? And so it was an interesting learning, in that I staffed up a growth team, tried to work on this metric, and then I realized, actually, like, I'm kind of just getting in the way. Like, this is really a thing that the core product team, um, most influences. Let me have a conversation with the core product leader and then shift that over to them. Um, so yeah. Just a super interesting experience.

    22. LRLenny Rachitsky38:45

        And then Chess.com.

    23. ACAlbert Cheng38:47

        The thing that's most unique about Chess.com is that they are super fanatical. Like, about chess. Uh, crazy.

    24. LRLenny Rachitsky38:54

        Makes sense.

    25. ACAlbert Cheng38:55

        That resonates. I mean, you shouldn't be surprised. Obviously, the name of the company is like this. Um, but they've always hired people from around the world. The company's always been globally remote. They just hire people that love chess. They play all day, they watch the streams. Our Slack is always blowing up with people's chess moves and games and whatnot. You know, I think... I want to say this a little bit delicately, but, like, Duolingo, even though the product they're providing is around language learning, I think the original ethos of how to start the company was really around motivation, right? The hardest thing to... It's habits, right? It's how do you build that daily habit? And I actually, in many ways, see language learning as, like, their first vehicle. And what they have a superpower in is that, again, the motivation, the habits, et cetera. So, that's kind of Duolingo. And Grammarly, actually, kind of similarly, right? Like, people know them for the spelling and grammar corrections, but what's really unique about them is they, they're integrated across tons and tons and tons of applications. There's not many, many products that work like that. That's really unique. And so now, if you hear, like, she shared their new CEO talk about, like, the AI superhighway and all that type of stuff, right? They can now use that technology to provide a lot more than just grammar writing. And so my point is just that, like, chess is about chess, 100%. It's in the ethos. People are crazy passionate. That just means we're always dogfooding the product. There's an, an en- just an amazing energy in the company to just use the product all the time, come up with ideas. And I love that environment. I think that's fun for me.

    26. LRLenny Rachitsky40:28

        That is so cool. And what I love about what you're saying is, there's no right or wrong answer.

    27. ACAlbert Cheng40:33

        Yeah.

    28. LRLenny Rachitsky40:33

        All of these companies are killing it. I think Duolingo is worth like, $10 billion, something like that, and keeps growing. Like, I'll look it up in a second. Uh, and Grammarly is worth a ton, and then Chess.com is doing super well. So, I think that's a really interesting takeaway here is, is you can succeed in a lot of different ways.

    29. ACAlbert Cheng40:51

        Yeah.

    30. LRLenny Rachitsky40:52

        What's really cool about Duolingo, I was just thinking as you were talking, is, yeah, it's just interesting that this very structured, methodical way of building is working so well. 'Cause you could listen to that and be like, "Ugh, that's, I don't want to work like, this rigid way." But the fact that it is killing it tells us this actually works really well. If you find something that works, lean into it.
11. 45:53 – 51:19

    How AI is changing Chess.com

    1. LRLenny Rachitsky45:53

       Okay. You talked about AI a little bit here and there, I wanna follow that thread. As a growth person, I imagine AI informs chess.com in a lot of way- ways.

    2. ACAlbert Cheng46:00

       It does.

    3. LRLenny Rachitsky46:01

       So there's kind of, like, two buckets here. How is AI changing the product, say at chess and other places you work, and then how is AI impacting your work as a growth person? So pick one or both buckets and, and share there.

    4. ACAlbert Cheng46:13

       Yeah, I'll, I'll tackle them in sequence.

    5. LRLenny Rachitsky46:15

       Sweet.

    6. ACAlbert Cheng46:15

       I'll start with the, the chess one, um, just 'cause I have maybe a slightly unique take on that one. So, chess and AI, they've been intertwined for almost a century, like some of the early computing pioneers, like, they just figured, "Yeah, chess is an interesting game. We can test machine intelligence and write some algorithms or whatnot." And then fast forward to, like, 1997, and you had IBM, they had their Deep Blue application, who actually, um, beat the, the world champion back then, which was Garry Kasparov. That was, like, a huge moment of, like, of shock and reckoning of like, "Oh, man, is AI gonna take over humans? Are we gonna have jobs?" And like, all this stuff, and this is, you know, 30 years ago. And thankfully, we're all still here and more people are playing chess than ever, right? And so the game of chess, and chess.com specifically, have learned how to augment, I guess, the human playing experience with the power of chess engines, which, you know, are definitely a powerful form of AI. It's not LMs, to be clear. But there's engines like Stockfish these days that are just dramatically better than the top grandmasters in the world. Like, um-

    7. LRLenny Rachitsky47:22

       Is that where we're at? It's just, like... I remember when it beat humans and now it's just dramatically better now.

    8. ACAlbert Cheng47:26

       It's dramatically better.

    9. LRLenny Rachitsky47:28

       Wow.

    10. ACAlbert Cheng47:28

        Yeah, I think... There's a rating system that compares, like, relative skill level, and an average chess player is somewhere like 1000, maybe a 1500 on the high end. A top grandmaster like Magnus Carlsen is, like, a 2800, and then Stockfish and similar engines are, like, 3600.

    11. LRLenny Rachitsky47:45

        Wow.

    12. ACAlbert Cheng47:45

        And so to put that in comparison-

    13. LRLenny Rachitsky47:47

        It's, it's cool it's not-

    14. ACAlbert Cheng47:47

        Yeah.

    15. LRLenny Rachitsky47:47

        Like, at least it's not 10,000 or a million. I don't even know if that's possible.

    16. ACAlbert Cheng47:51

        No, it's, it's not 10,000, but it's similar to, like, if the chess engine was playing without a major piece, like a rook or something, they would still-

    17. LRLenny Rachitsky47:57

        Wow.

    18. ACAlbert Cheng47:57

        ... be competitive against the best players, so.

    19. LRLenny Rachitsky47:58

        And this is the ELO score? Is that the term?

    20. ACAlbert Cheng48:00

        Yeah, the ELO score, ELO rating.

    21. LRLenny Rachitsky48:01

        Okay. Magnus is, is what you said, about 2800?

    22. ACAlbert Cheng48:04

        Yeah.

    23. LRLenny Rachitsky48:04

        And, and then the Stockfish is, would you say 3600? 38-

    24. ACAlbert Cheng48:08

        Yeah, and, and really it's like, it's because computing power is so amazing and there are so many techniques for how to do, like-

    25. LRLenny Rachitsky48:14

        Yeah.

    26. ACAlbert Cheng48:14

        ... deep evaluation on specific chess lines, they can calculate tens of millions per second. So it's not realistic (laughs) for, for a human to compete against that. But yet, like, watching some of these chess engines play has opened up a lot of creativity and new strategies, new lines, new appreciation for the game. And our chess.com approach is that we can bring this technology for every user, even people that have never moved a piece before. I talked earlier about that game review product, that's exactly what this does. So behind the scenes, we're running chess engines t- to basically spit out evaluations for every move that you make, and then we translate that and make that approachable to the user using, you know, their native language and play an approachable, uh, style, right? And even with audio and things like that as well. And that part of it, like the personality, the speech back to the user, that part is LMs.And so I guess my point is that, again, chess and AI have been intertwined forever. But for us, what's most important is that we keep the customer at the north star of it. We're not just supplying LLMs just 'cause it's the new hot thing. You gotta apply the right technology for the right, you know, feature to provide value to the user. And so we ne- we try not to ever lose sight of that and get... not let, let hype get us too carried away.

    27. LRLenny Rachitsky49:31

        It's just sh- really surprising. I think people would not have expected AI can now beat every human alive, ever. And we s- and chess is at an all-time high. People want to keep playing and are playing more and more than they've ever played.

    28. ACAlbert Cheng49:44

        Yeah.

    29. LRLenny Rachitsky49:44

        Not an expected timeline we're on.

    30. ACAlbert Cheng49:45

        And, you know, interestingly, LLMs themselves are quite bad at playing chess. Like, they hallucinate moves. They look at patterns, right? They're, they're very good at pattern recognition but not so good at going super, super, super deep on a specific chess thing. And if you've even tried to, like, create ch- or look at chessboard images on (laughs) ChatGPT, a lot of them have the wrong number of squares, they're not set up properly. And so, you know, I don't want to be too dismissive. I'm sure it's gonna get much stronger at, at reasoning. And actually, Google recently sponsored a tournament where all the top LLMs played a tournament against each other. Um, so that was pretty fun to watch. They're improving, but, you know, this chess is specifically a game that having a, a trained, you know, deep, deep computing engine is just gonna be much, much, much more powerful than LLMs.
12. 51:19 – 53:47

    How AI is changing the growth role

    1. ACAlbert Cheng51:19

       damn good.

    2. LRLenny Rachitsky51:19

       Okay, let's go back, back on track to where we were going.

    3. ACAlbert Cheng51:22

       (laughs)

    4. LRLenny Rachitsky51:22

       So this was how AI is impacting chess.com. How is AI changing the, just the work of a growth person?

    5. ACAlbert Cheng51:30

       I like to describe growth as the, the job is to connect users to the value of your product. And in order to do that, um, what I like to do is think about that user journey, again, and essentially staff teams that are oriented around each element of that user journey. And those teams have specific metric goals, they have roadmaps, et cetera, and then they go run against them. So that's, like, how it's structured. Um, AI, I think can be applied to speed up some elements of that exce- essentially experiment cycle that you get through. So one example is in product discovery. As opposed to core product, which tends to have longer time frames, and you might do, like, you know, thorough user research or market research, you know. It's more, it's more foundational, more from first principles, et cetera. Growth is a little bit less like that. It's like you're running a lot of experiments, and you're... The output of any given experiment is the input, like, to your next idea, right? And so historically, I don't even, I don't even mean historically, but just a few months ago, right? Like, we were operating in a... (laughs) It's got kind of history, I, I suppose.

    6. LRLenny Rachitsky52:36

       Yeah, yeah.

    7. ACAlbert Cheng52:37

       Um, but, you know, there would be a lot of manual writing of these, like, analyses docs. You'd have to read them, you'd have to understand what insight you want to kind of grab from them, and then write another spec to, to translate that idea. That's still happening to some degree, but I think that's a spot where even tools like ChatGPT are super helpful, right? You can just plug in, like, a, an analysis that another person wrote and just have it summarize for you and give you advice on, you know, ideas to go try. And so that ideation, that research cycle is much, much faster. I talked a little bit about prototyping also just becoming much, much faster than before. We have not yet gotten to the point where, like, product managers themselves are actually shipping the code into production. But it's dramatically shortened the amount of time it takes to conceive of especially, like, a bolder idea that you might have. And so when I talked earlier about explore and exploit, right? A lot of the explore was harder to do, but now it's a little bit easier to do. You can take a, a broader concept and visualize it. And when you can visualize it, center it around the team, get people to click around it, that makes a world of difference. Um, so those are just a couple examples

13. 53:47 – 57:22

    Tips for running successful experiments at scale

    1. ACAlbert Cheng53:47

       that come to mind.

    2. LRLenny Rachitsky53:47

       Awesome. I want to go back to this phrase right at the beginning of this answer that you shared that I think is really helpful, that you see growth as simply your job is to connect users to the value of your product.

    3. ACAlbert Cheng53:58

       Yeah.

    4. LRLenny Rachitsky53:59

       Can you speak more to that? 'Cause I think that's such a nice way of clarifying what is growth's role.

    5. ACAlbert Cheng54:04

       Yeah. I, it's resonates deeply with me because I, I feel like growth sometimes gets this, um, reputation, I guess, that it's just pure, like, metrics hacking. Like, we're cold, cold people that just are trying to move a particular metric up, and we're gonna do whatever we can to, like, you know, throw walls and paywalls and add friction in those, all these spots. Um, and even though that could theoretically work in, at, like, a micro level on a specific feature or a specific metric, I think what's most healthy for a company, and, you know, I want to work at durable companies, right? Is to think about the user holistically, right? And when you take that framing of connecting, um, users to the value of your product, that value can change for a user over time. And that also lines up really nicely to the journey, right? Like, what a, uh, someone that's not even a user yet needs to understand about the value proposition is super different than what a habitual user of three-plus years, uh, might need, right? And so the teams working on them should think from that perspective. And then from there, right, then ladder into, like, specific problems to solve, hypotheses, et cetera.

    6. LRLenny Rachitsky55:15

       Following that thread a little bit more-People listening to this are imagining, "How do I get better at experimentation? How do I run more experiments? How do we do this better?" What are two or three tips and best practices that you think peoples need to hear, maybe are not totally aware of when they think about getting better at experimentation on our teams?

    7. ACAlbert Cheng55:36

       I think the first thing is just start somewhere. You know, I- I-

    8. LRLenny Rachitsky55:40

       (laughs)

    9. ACAlbert Cheng55:40

       ... just read this Atlassian, uh, state of product report, and it was like 40% of product teams, like, basically don't run experimentation at all. And there may be some good reasons for it. I mean, it could be philosophical or maybe you're more, you know, B2B oriented or whatever. So I- I get it. But I think for a lot of, especially if you work on a consumer product that has some degree of scale, some degree of frequency with your product, you can collect enough data. And also, I have found, you know, I can pattern match all day long, I've worked at a lot of companies, right? But I'm wrong all the time, and I think consumer behavior can be very fickle. And especially when you work at a company, you become a power user naturally. So sometimes you, you may forget, like, what the actual user experience is for a brand new user. And so you leave a lot of opportunities on the table if you don't even try to experiment. So I just encourage taking that first step. Just run an A/B test. Find a third party tool or something that you can integrate quickly and, or- or even just work with your engineers to spin something up. Just get in the practice of, you know, crawled and walked and run type of thing.

    10. LRLenny Rachitsky56:40

        Do you have a favorite tool, by the way? Just to throw out, is there like a go-to tool for you?

    11. ACAlbert Cheng56:44

        We used, uh, Statsig at Grammarly, and I saw that they recently got acquired, so that was exciting news. Duolingo and Chess.com both have an in-house experimentation approach.

    12. LRLenny Rachitsky56:54

        Cool. Sweet.

    13. ACAlbert Cheng56:54

        Um, pros and cons to either. Obviously, Duolingo is an experimentation machine and so it's been, uh, a huge accelerant to have our own thing specifically tailored to, uh, to be excellent at that. But, uh, no, I- I typically don't encourage companies to build experimentation in-house from day one. You know, at a certain scale, it can make sense. And some of these companies, right, they were started 15 years ago when these tools weren't out, so it was just something they

14. 57:22 – 1:01:19

    How to shift company culture toward experimentation

    1. ACAlbert Cheng57:22

       had to do.

    2. LRLenny Rachitsky57:22

       Something that you mentioned to me, uh, at Chess.com, your goal is to run a thousand experiments a year. You said you were at 250. Talk about just that as a, as a north star.

    3. ACAlbert Cheng57:32

       Yeah, so part of having team members that are fanatical about chess is that the company can get pretty damn far just like building for themselves, building for the community, and not actually being very experimentation and data oriented. The problem with that is that you can have relatively lumpy growth, right? And so part of the- the kind of excitement of me joining the company was to help smooth that out and bring in that experimentation mindset. So prior to 2023, the company practically didn't experiment at all. Last year they did about 50. This year they're on pace for about 250. And then next year we have that ambitious target of 1,000. Did I make it up? Yes, absolutely, I made it up. But (laughs) but it's still a target and, uh, a thing for the teams to- to think about. And 1,000 experiments by itself, like, if you just did that but you didn't learn, you didn't make an impact, then that's kind of a waste of time, right? The whole point of setting a goal is that you can have conversations about what would need to be true to actually hit that goal. And so that leads to, um, insights like, actually, we need not just product management or- or engineering to be running these experiments. Um, we can experiment with lifecycle marketing, changing copy of push notifications and emails. We can experiment with app store screenshots and, you know, keywords and stuff like that. The way of all sorts of content marketing teams, et cetera, right? We could have engineering enable no code for specific screens. Think about our home screen or our pricing screen where we might wanna do a lot of just tests that are configurable without engineering support. Um, we might wanna just, like, track our progress and- and look at it from time to time and make sure that we have the right, you know, observability around this. So anyway, that's the stuff that really matters as opposed to the, you know, hitting that goal, um, itself. So don't tell the team, but I don't actually care that much if we actually hit 1,000. But, uh, I think if we get pretty close and we accomplish some of these things, we'll be in really good shape.

    4. LRLenny Rachitsky59:26

       Okay, we'll make sure none of them watch this.

    5. ACAlbert Cheng59:28

       (laughs)

    6. LRLenny Rachitsky59:29

       Um, I think Chess.com is, and this ex- this is just a, such a cool example of a culture shifting dramatically from zero experiments to, sounds like two years later, 1,000. Which is like three a day. Like, you know, that's, you know, there's many teams running experiments in parallel but that's a lot. What has helped you most shift that culture? Is it just the CEO being like, "This is the way we're gonna go"? Uh, what- what have you learned about helping shift a culture from, "No, we're not doing experiments" to 1,000 experiments a year?

    7. ACAlbert Cheng59:58

       Yeah, I mean definitely a lot of credit to the- the CEO and co-founders like Eric and Danny, they're amazing. It's not their intuitive way of thinking about growing companies, but their mental flexibility and encouragement, right, to- to evolve and add this as a- as a tool for the company has been awesome. And they've been on the front lines preaching product-led growth and experimentation just as much as- as I have. So I'm glad that you brought that up, because I think that is critically important. For me joining a company to not be at odds with, you know, the- the co-founders and the existing approach of the company. I think that's absolutely, absolutely critical. I think the, you know, I started this podcast with the example of the game review and the positivity and how that was shared. I mean, I think those types of things are really what motivate people, right? They need to see this working in practice.

    8. LRLenny Rachitsky1:00:45

       Wins.

    9. ACAlbert Cheng1:00:46

       You can... Yeah, you need wins, you gotta celebrate them, people feel good about the learning, it's applied across the board. Like, who's not gonna be energized by that I think, right? So, um, you can't just set goals in a vacuum and, you know, create it from- from top, right? People have to see it working and- and when it works, like the metrics move and you learn faster and you ship faster and that- that's a, that's a great environment to be a part of.

    10. LRLenny Rachitsky1:01:07

        What was the first experiment you guys ran? Do you remember?

    11. ACAlbert Cheng1:01:10

        I don't know.

    12. LRLenny Rachitsky1:01:11

        (laughs)

    13. ACAlbert Cheng1:01:11

        Before my time actually. (laughs)

    14. LRLenny Rachitsky1:01:13

        Okay, okay, got it. So they were already going down this track before they brought you in.

    15. ACAlbert Cheng1:01:16

        They had run, they had run a, they had run some.

15. 1:01:19 – 1:04:41

    Key lessons from running experiments at scale

    1. ACAlbert Cheng1:01:19

       Yeah.

    2. LRLenny Rachitsky1:01:20

       Okay. Sweet. Are there any other key lessons that you think people need to know to...... be successful running experiments at scale?

    3. ACAlbert Cheng1:01:30

       The system matters just as much as any given experiment, probably even more, right? I think starting with a growth model so you have an understanding of how your company grows in the first place and which channels you're going to leverage is critical. You need to make sure that you are instrumenting your product in and out (laughs) . Otherwise, you're gonna run experiments and have wonky results. Um, I won't name which company, but I, I was, I was part of a company that had an in-house experimentation tool. About three months into the company, we're running, like, some experiments, and we realized that user retention was actually configured backwards. (laughs) So all positive results were negative results.

    4. LRLenny Rachitsky1:02:09

       Oh, geez.

    5. ACAlbert Cheng1:02:10

       Um, so that was kind of embarrassing. And, uh-

    6. LRLenny Rachitsky1:02:12

       You just go-

    7. ACAlbert Cheng1:02:13

       ... that will never happen again.

    8. LRLenny Rachitsky1:02:13

       ... through and undo all those experiments and just-

    9. ACAlbert Cheng1:02:16

       Yeah, I was like, "That-"

    10. LRLenny Rachitsky1:02:16

        ... drive up retention.

    11. ACAlbert Cheng1:02:17

        "... it's kind of weird. Like, we're seeing people use the, the features a lot more. Why is the user retention-"

    12. LRLenny Rachitsky1:02:21

        Wow.

    13. ACAlbert Cheng1:02:21

        "... going, going negative?" Um, so I have plenty of horror stories around that type of stuff, but-

    14. LRLenny Rachitsky1:02:25

        Oh, my God.

    15. ACAlbert Cheng1:02:26

        ... yeah.

    16. LRLenny Rachitsky1:02:26

        On the flip side of horror stories, you've shared a bunch of cool examples of experiment wins. Is there another that comes to mind of one you're really proud of or that was really trajectory changing, either at Duolingo or Grammarly or, or Chess?

    17. ACAlbert Cheng1:02:38

        The... So I already shared one of chess.com and one of Grammarly. I mean, I could talk a bit about, about Duolingo as well.

    18. LRLenny Rachitsky1:02:44

        Mm-hmm, yeah.

    19. ACAlbert Cheng1:02:45

        Um, Duolingo, uh... And you had Jackson on the podcast, right? When we talked-

    20. LRLenny Rachitsky1:02:50

        Yes.

    21. ACAlbert Cheng1:02:50

        ... about the streak.

    22. LRLenny Rachitsky1:02:51

        Yes, talked about streaks.

    23. ACAlbert Cheng1:02:52

        Uh, so I-

    24. LRLenny Rachitsky1:02:52

        Those great streets.

    25. ACAlbert Cheng1:02:52

        ... also don't want to steal his thunder, because I was gonna think about that, but the amount of learning through, um, commitment and putting streaks on a calendar and just getting people started, right? As opposed to achieving some large milestone, um, that was huge. Uh, I think we, we did something interesting. We spun up a virality team, and virality is this, like, really amorphous thing to me. I think it's really hard to generate virality in your product, but Duolingo is a product that is shared quite a bit. And so we invested, actually, in some time to, um, essentially add screenshot tracking for, like, a brief period of time in the app, just so we could find out the hotspots of where users were doing screenshots. And you see this in other apps, too. It's not necessarily, like, you know, so- some horrible thing. But we, we did this for some period of time, and we were able to basically articulate and say, "Okay, um, you know, streak milestones were the obvious one, really funny challenges that you get in the Duolingo experience is also super highly shared. Um, advancing in the top three of a leaderboard is another thing." Anyway, so you can find these different moments where that's the case, and then we staff those moments with illustrators and animators, and created these really delightful experiences around them, and I... That worked amazingly well. So as opposed to going against, I guess, human intuition and trying to get them to share stuff that they otherwise wouldn't, on the margins, want to share, like, lean into it more. Actually, like, grab the moments where users are already organically screenshotting, and make those much, much, much better, and you can kind of 5X or 10X and, and drive a lot of growth that way, too. So that's not so much an experiment. That's more a core product thing, but, uh, you know, it just resonated with me that that was interesting.

    26. LRLenny Rachitsky1:04:33

        Well, it connects to your explore and exploit methodology. Just find where, explore where things are happening and then try to exploit in a nice, positive

16. 1:04:41 – 1:07:50

    The three pillars of successful gamification

    1. LRLenny Rachitsky1:04:41

       way.

    2. ACAlbert Cheng1:04:41

       You got it.

    3. LRLenny Rachitsky1:04:42

       Speaking of that, you've mentioned this with Duolingo, is just very good at habit formation and motivation, behavior. Feels like Chess is good at this, too.

    4. ACAlbert Cheng1:04:51

       Mm-hmm.

    5. LRLenny Rachitsky1:04:52

       You've worked at both these companies. What have you learned about how to motivate people, how to create habits?

    6. ACAlbert Cheng1:04:57

       Again, like, Duolingo would not have started without this, uh, insight from day one, right? They, they aim to, to focus on motivation and build a lot of these, like, tactics. Um, Jorge actually had this model of, like, gamification, uh, patterns having essentially three pillars to it. You have the core loop, you have the, uh, meta game, and then you have the profile. And so we actually thought about it that way, too, where, you know, your core loop is, is your lesson that you go through. You do a lesson. You get some rewards. You extend your streak, and then the next day, you get a push notification. It's kind of the core loop of the product. And making that really tight is, is super important, 'cause people need a habit to stick to. Then you need a meta game, which for Duolingo is kind of like the path, but it's also the leaderboard achievements, kind of long-term things that you're gonna strive to, such that you have, like, long-term, I guess, motivation, uh, to continue doing the thing. And then the profile is also critical, because you build up a profile over time. It's a reflection of your investment inside the product experience. And so when you nail those three things, you can end up with a long-term learning journey that can be quite successful. And then to flip over to the chess.com side, like, what we see is that over 75% of our new users, they classify themselves as like, "I'm completely new to Chess, or I'm a beginner." And unfortunately, if you're new to Chess and you're a beginner, you're not gonna have that fun of a time playing live games. We see this in the data. It's like, less than a third of those users actually win their first game. And when you lose a game, user retention is 10% worse than when you win a game.

    7. LRLenny Rachitsky1:06:28

       Hm, that's not so bad, but, eh, at scale, that's bad.

    8. ACAlbert Cheng1:06:31

       Yeah, and, and it could be worse. That's true, but... And so typically, what, like, a lot of mobile games will do is they'll just create, like, a super simplified version of the game. It's harder for us to do at Chess.

    9. LRLenny Rachitsky1:06:41

       (laughs)

    10. ACAlbert Cheng1:06:42

        And so without changing the rules of that, right? I think that, that, uh... I don't know. It's just very eye-opening to me that when you're trying to learn something, whether that be language learning or, or chess or whatever, um, usually those first steps are fraught with, you know, a lot of self-doubt and reinforcement that you're not good at the thing. And so it's, it pays to be very intentional to craft experiences that, you know, guide the user around that.

    11. LRLenny Rachitsky1:07:09

        Well, I can't help but ask, is there anything that helped that a- along?

    12. ACAlbert Cheng1:07:12

        Yeah, so, like, something we're experimenting right now is just, like, purely if you say that you're new to Chess, we're gonna craft a more delightful learn how to play experience, as opposed to dropping you into a live game. That's an example. Another is, like, hiding your ratings for the first five times such that you're not seeing your rating kind of plummet. Um, so there's a lot of tips and tricks you can do.

    13. LRLenny Rachitsky1:07:29

        I'm just imagining a little guy that's like, "Here's how you win, shh." (laughs) "Here's how other people s-"

    14. ACAlbert Cheng1:07:32

        Yeah, or play, play against the coach, play against a friend, play against a bot. There's, there's a bunch of different avenues you could take on.

    15. LRLenny Rachitsky1:07:38

        Well, what I'd love is play against someone real and then here's, like, tru- here's where you should move. Just like, "Here. Here's- we're gonna let you- we're gonna help you win."

    16. ACAlbert Cheng1:07:44

        Oh, like, uh, like, like a hint in real time?

    17. LRLenny Rachitsky1:07:46

        Yeah.

    18. ACAlbert Cheng1:07:46

        Yeah.

    19. LRLenny Rachitsky1:07:46

        Yeah, yeah.

    20. ACAlbert Cheng1:07:47

        Well, I don't, I don't wanna be playing you, then.

    21. LRLenny Rachitsky1:07:48

        (laughs)

    22. ACAlbert Cheng1:07:49

        (laughs)

17. 1:07:50 – 1:10:38

    The most counterintuitive lesson about building teams

    1. ACAlbert Cheng1:07:50

    2. LRLenny Rachitsky1:07:50

       Okay. Let me ask you a couple more questions. One is just zooming out a little bit. What's, what's the most counterintuitive lesson you've learned about building products or building teams across the many companies you've worked at?

    3. ACAlbert Cheng1:08:03

       Yeah. I've talked a lot about products, so maybe I'll flip to the team side for a bit. Um, I think the standard way to hire and build a team is, you know, you fill out a JD. It's got a whole bunch of different characteristics that you're looking for. You typically will find, you know, a short list of companies that are kinda similar to yours, and then you try to hire for that, right? I think that's kinda the, the typical default path that, that a lot of companies take. And I was really struck by, you know, my experience working at, you know, some smaller startups or, you know, take Duolingo as an example, where, um, over and over and over, like, I saw some of the highest performers just being people that had very high agency, had that clock speed, had that energy. Um, yes, they, they cared about the mission, but they didn't necessarily need to have deep experience on that matter. And in fact, sometimes that experience could be a crutch in, in certain ways, especially in this world where the grounds are shifting so fast with AI. A lot of your, like, learned habits actually need to be intentionally discarded. You know, you need to have a beginner's mind on this type of stuff. So I think this is more true than ever, um, like, looking for people that respond and move quickly and, and think, um, you know, just, just faster and move faster, right? I think speed, the fastest speed of learning, uh, those types of companies are the ones that I wanna bet on, 'cause I think those, those will end up, uh, surviving and thriving.

    4. LRLenny Rachitsky1:09:25

       So just to double-click on this, (clears throat) this idea of high agency is very, uh, trending these days, of just, like, hire high agency people. To unpack that a little bit, you mentioned a few of these traits, so let's just help people see what you see. So one is clock speed, just they think fast, they move fast, uh, they learn fast. What else, what else do you look for that helps you see that they're high agency people?

    5. ACAlbert Cheng1:09:49

       Yeah. I mean, a lot of it actually happens outside of the interview process, interestingly. (laughs) So (laughs) a lot of it is, uh, you know, the types of questions they ask. Have they actually tried your product and gone deep into it? A lot of it is the, you know, it's the references. It's the, like, communication that they have to even set up your interview. Like, it's the energy they bring into the conversation. You can actually pick up a lot of soft signals on some of these traits. Um, yeah, over time, you kinda pick up on some of these patterns. I don't know that I'm perfect at it, but I've, I've learned to balance those things quite a bit more than I did in the past when I would just purely read from my questions and my rubric and not care about anything else.

    6. LRLenny Rachitsky1:10:27

       Yeah. There's, like, a vibes component to it. This is also kind of support for the work trial way of interviewing versus just a talk interview where you have them actually work with you for a week or whatever.

    7. ACAlbert Cheng1:10:36

       That's a great point.

18. 1:10:38 – 1:13:28

    Deciding what size company is a good fit

    1. ACAlbert Cheng1:10:38

    2. LRLenny Rachitsky1:10:38

       Okay. Uh, one other question I wanted to ask you. You've worked at a bunch of different sizes of companies, from startup to Grammarly. I don't know if it's, you call it a big company-

    3. ACAlbert Cheng1:10:47

       (laughs)

    4. LRLenny Rachitsky1:10:47

       ... bigger company. Duolingos. Duolingos, I don't know, how big is Duolingo? It's like thousands or-

    5. ACAlbert Cheng1:10:51

       They're about 1,000 people.

    6. LRLenny Rachitsky1:10:52

       Okay, cool.

    7. ACAlbert Cheng1:10:52

       Yeah. But I wo- I worked at Google, too, to start my career, so yeah.

    8. LRLenny Rachitsky1:10:55

       Oh, right. Okay. What have you learned about just the size of company that makes you happy? What have you learned about just helping other people that you talk to decide what size of company is good for them?

    9. ACAlbert Cheng1:11:05

       I, I definitely believe that everyone has a company stage that they shine best at. I've personally gone through this journey of big tech to, like, tiny, tiny, tiny startup, then landed in the middle, which I consider, like, my own Goldilocks zone. I talked earlier about, like, what actually gives me, personally, a lot of energy is seeing across a company's efforts, but also the company being small enough that I can get into the details. I can work with the specific teams. I can read, experiment, you know, results. I can look at the pixels. And so I find that the balance of those two things tends to fit best with medium-sized companies. But that's me, right? I, I think at big companies, uh, like a Google, you're dealing with immense scale, which is interesting by itself. You learn a lot of best practices from your peers. They have all the kind of tools and functions that you would possibly want to go learn from. But they can tend to move slower, and it's harder to kinda ship things and get them out the door, which, you know, eventually drove me nuts a little bit. On the flip end of the spectrum, these tiny startups, they move incredibly fast, but I grew, like, all my gray hair from those tiny startups (laughs) because no one knows about your company. And so you're recruiting people one by one. You're r- you know, trying to get users one by one. So yeah. You can, you can learn fast and ship a lot of things, but if you're trying to make a big impact on the world, it can be, uh, be actually pretty grueling to do so at, at really, really, really small startups. Now, some of them do hyper scale and make it out. And, um, obviously, I- I'm not one to, to trash that 'cause that's the path that I tried for, for quite a while. But for me, like, I really like the zone where I can contribute at scale but also execute at a pace that's more on, like, the daily and weekly scale, right, as opposed to monthly and quarterly.

    10. LRLenny Rachitsky1:12:50

        And when you say medium, what size of company is that, roughly?

    11. ACAlbert Cheng1:12:53

        Yeah. So, um, these companies that we've talked about on the podcast are about 500 to 1,000 people. Typically, these companies have all been around, let's say, 10 to 20 years. Like, they're durable, ideally profitable, um, they have a good leadership team. But there's still a lot of dimensions to go figure out. A lot of them are in, in key inflection points. Um, so they're certainly not stagnant, right? You, you need to find a place that's dynamic too.

    12. LRLenny Rachitsky1:13:17

        Interesting. 10 to 20 years old. Uh, I don't know. That's a, that's a c- not many people would feel like that's where I wanna be.

    13. ACAlbert Cheng1:13:24

        Yeah.

    14. LRLenny Rachitsky1:13:24

        Uh, I love that you found a number of companies like that that you enjoyed working at.

19. 1:13:28 – 1:16:42

    Failure corner

    1. LRLenny Rachitsky1:13:28

       The last question-And this is gonna be taking us to a recurring segment on the podcast that I call Fail Corner. People hear all these stories of all these experiments and all these companies you worked at. They're all killing it up and to the right. Uh, in reality, you've touched on this, a lot of things don't work out great. So, can you share a story when something went wrong, when you failed, and what that taught you?

Episode duration: 1:25:24