Failure

1. 0:00 – 3:25

   Lenny: Why I’m focusing on failure

   1. LRLenny Rachitsky0:00

      (instrumental music) Today, we've got another very special compilation episode. Something that I've been pulling on more and more with the podcast and the newsletter, in case you've noticed, is failure. Normally, I spend a lot of time researching how the best companies and the best product leaders operate, but you can learn a lot, and often a lot more, from failure. And so what we've done with this episode is we've looked at all of the past episodes we've done and pulled out all the most interesting and insightful stories of failure, and turned it into this very focused episode on failure. I hope that you find this useful and interesting. Let us know what you think in the comments on YouTube or on LennysNewsletter.com, or just let me know on Twitter. If you like this, we'll keep doing this. If not, we'll probably not. Either way, I hope you enjoy. Before we dive in, here is a short word from our wonderful sponsors. (instrumental music) Let me tell you about our product called Sendbird, the all-in-one communications API platform designed for both web and mobile apps. In a world saturated with multi-channel communication, product teams are discovering the effectiveness of in-app communication. With Sendbird, businesses can elevate their in-app experience with decluttered and branded communication featuring AI-powered chatbots, one-way messages, chat, video calls, and live stream capabilities, all tailored for commerce, marketing, and top-tier support. Forward-thinking companies such as Hinge, Patreon, Yahoo, Accolade, and more use Sendbird to build in-app communication experiences that drive engagement, conversion, and retention. In-app communication has the highest conversion, highest engagement, and highest satisfaction of any communication channel. And when it comes to investing in this channel, trust Sendbird to take your in-app communication experience to the next level. Start today with Sendbird's free plan, and as a listener of Lenny's podcast, you'll get an additional two months of unlimited usage and access to all premium features, including creating your very own generative AI chatbot. Visit sendbird.com/lenny to begin your free journey. That's sendbird.com/lenny. This episode is brought to you by Eppo. Eppo is a next generation A/B testing and feature management platform built by alums at Airbnb and Snowflake for modern growth teams. Companies like Twitch, Miro, ClickUp, and DraftKings rely on Eppo to power their experiments. Experimentation is increasingly essential for driving growth and for understanding the performance of new features, and Eppo helps you increase experimentation velocity while unlocking rigorous deep analysis in a way that no other commercial tool does. When I was at Airbnb, one of the things that I loved most was our experimentation platform, where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Eppo does all that and more with advanced statistical methods that can help you shave weeks off experiment time, an accessible UI for diving deeper into performance, and out-of-the-box reporting that helps you avoid annoying prolonged analytic cycles. Eppo also makes it easy for you to share experiment insights with your team, sparking new ideas for the A/B testing flywheel. Eppo powers experimentation across every use case, including product, growth, machine learning, monetization, and email marketing. Check out Eppo at geteppo.com/lenny and 10X your experiment velocity. That's geteppo.com/lenny.

2. 3:25 – 8:03

   Katie Dill: The single meeting that changed how Katie leads forever

   1. LRLenny Rachitsky3:25

      All righty. First up, we've got Katie Dill, who is head of design at Stripe and former head of design at Airbnb and Lyft, sharing this amazing story of how the entire design team at Airbnb basically rebelled against her soon after she joined, and what she learned from that experience.

   2. GSGuest (Airbnb leader sharing an intervention story)3:42

      I'm happy to talk about it, because frankly, it was the biggest learning experience of my leadership career, or at least that happened in one moment, and it happened in my early days at Airbnb. So, I was hired to take on the design organization, or the, the experience design organization. That's basically the product design team, uh, which was 10 people at the time. And so they had been reporting directly to one of the founders, and they were gonna start reporting to me. And during my interview process, I learned a lot about what was working and what wasn't working, and some of the, you know, trials and tribulations with the, the design organization and its collaboration with others. So, it seemed like there was room for improvement in how engineering, product management, and design all worked together, and there was also really low engagement scores in the design team. And so I kind of came in ready to go and, you know, excited to try to help make some change based on all the things that I had learned, uh, from various leaders and people across the company. And I, you know, came in swinging, ready to go. And then about a month into my time there, I got a meeting on my calendar, Thursday, 8:30 AM. It was an hour and a half with, uh, half of the design team, so that was five people, and our HR partner.

   3. LRLenny Rachitsky4:58

      Oh, no.

   4. GSGuest (Airbnb leader sharing an intervention story)4:59

      It's not usually a good time.

   5. LRLenny Rachitsky4:59

      That's never a good sign.

   6. GSGuest (Airbnb leader sharing an intervention story)5:01

      Yeah. Uh, and I, I remember this so vividly. I remember walking into the office and, you know, all the rooms in Airbnb's office are very unique spaces that look like Airbnbs, but, of course, this was the one room with, like, all white walls and just, like, a gray, you know, flat rectangle table. And I walked into the room and there were five of them seated around the table, and they had a pack of papers in front of them, and they went on taking turns, you know, quietly reading from the papers all the things that they w- saw that I was doing wrong, and all the things that they didn't like about me, and, you know, (laughs) it was a really hard moment, you know? Their... I, I went through all the usual kind of, like, stages of grief when one hears feedback, which is just, like, you know, immediate want to, like, respond, to be like, "No," like, you know, "Well, there was a good reason for that," and, like, "That's not how it actually was," and, "This is why I did that."

   7. PAPaul Adams5:59

      But luckily, I had, you know (laughs) thank goodness I had the sense to just, like, listen and not respond in that way. I mean, clearly what they were telling me is that, you know, that was one of the things that was missing. And so, I, I heard them out and, you know, took it all in and, you know, regardless of each individual thing, what was very clear was that, you know, the missing piece, the theme that was across all of that is that I hadn't earned their trust. So, you know, whether, you know, how right or how wrong what I was doing was is like the key piece is that I wasn't bringing the team along with me. They had no idea that they could trust in, you know, what I was trying to build and what I was trying to shape and that I, I cared about them and that I had, you know, their best interest and shared goals at heart. And that was absolutely my fault. And in retrospect, you know, as hard as that was, I'm, like, very grateful and very amazed that, you know, that they could kind of come together and, and, and share that with me. Uh, it's, it can be hard to, you know, bring, you know, feedback forward like that. And so, it was an extremely valuable learning experience and I took from that to then sh- immediately shift how I was operating and really a key part in building trust was to, to listen. To hear out, you know, what, you know, the individuals on the team were, you know, setting out to do, what they cared about, uh, what motivated them. And so, I started to make pretty fast change and still moving in the direction that, you know, was necessary for the org to make like the, you know, the really large impact in how we were operating but bringing folks along with me. 'Cause you can inflict change on people, but if you want to do it with them, you, you really, you know, trust is the, the key element there. And then, you know, a couple months later we had the best engagement scores in the company. So like, it actually, like-

   8. LRLenny Rachitsky7:42

      Oh, wow.

   9. PAPaul Adams7:42

      ... you know, it did, it objectively improve the situation and I've, I've, you know, since then taken that on into next steps in other companies that I've joined and just think about like instead of coming and swinging, you know, come in listening so that you can really, you know, set out to make change that actually has, like, true positive impact on, on the folks around you and that you bring along

3. 8:03 – 18:38

   Paul Adams: Freezing onstage in front of thousands of people

   1. PAPaul Adams8:03

      with you.

   2. LRLenny Rachitsky8:03

      (air whooshing) Next up, Paul Adams, Chief Product Officer at Intercom, sharing the nightmare experience of freezing on stage in front of thousands of people, having to walk off stage and then people hearing him curse because the mic was still on, and then how he recovered, plus a few stories of building some of Google's most infamous product failures.

   3. PAPaul Adams8:23

      Yeah. Some- some things that happen in work are very memorable at the time and they don't really scar you. Uh, this goes in the book that have scarred for life. Uh...

   4. LRLenny Rachitsky8:30

      (laughs) .

   5. PAPaul Adams8:31

      Yeah. I, I-

   6. LRLenny Rachitsky8:32

      A little bit of

   7. GOGuest (brief laugh only)8:32

      (laughs) .

   8. PAPaul Adams8:32

      ... to cut a long story short, I was at Facebook just over a decade ago. Loved it at the time, um, I think it was a great place to be at the time, and, uh, based in San Francisco. I did a lot of talks for Facebook internally and externally. Um, Facebook had a keynote slot, always have a keynote slot at Cannes, the world's biggest advertising festival. And, um, the year prior, Zuck had been interviewed. He was the speaker, he'd been interviewed, he'd gotten a hard time on privacy. Uh, it didn't go well, as well as they'd hoped. So the next year, they asked me to do it. Maybe it was the Irish accent, you know, that, that-

   9. LRLenny Rachitsky9:05

      (laughs) .

   10. PAPaul Adams9:05

       ... made, made the offer come my way. Um, and yeah, I got out- I got out in front of the stage, you know, the world's, uh, biggest advertising stage, and I'd say I was like three, four minutes into the talk, a talk I'd given, a very similar talk to one I'd given lots of times, and, um, I just froze. I, I couldn't remember what I was supposed to say. Uh, it was the first ever time in my life I'd rehearsed a talk word for word. You know, us- usually, like, I have talking points and I'll ad lib and, you know, things get mixed around and it's kind of informal. This was like, you know, media trained, like, don't, do not say the wrong thing kind of talk, and I just could, could not remember what to say. I had some version of a panic attack. Uh, walked off stage. I was still mic'd up, uh, cursed. Everyone started laughing. I was like, "Geez, are they laughing at me?" You know, "Oh my God, this is..." Um, but I, I, I managed to turn around. I walked back out. I'd kind of been disarmed internally in my head and the rest of it went well. But it was, uh... And I was famous that night. You know, out in Cannes afterwards, like, on the, on the, whatever, the, the sea front, it's just, like, rosé everywhere. And, um, yeah, I was famous and infamous for my performance.

   11. LRLenny Rachitsky10:13

       I feel like you lived the worst nightmare that everybody has when they're thinking about giving a talk, and I think what's interesting is you survived, and I think that's a really interesting lesson is, like, you could freeze in front of thousands of people, walk off stage (laughs) , and then it works out okay.

   12. PAPaul Adams10:30

       Yeah. And- and it all happened kind of, uh, organically I guess, or very naturally, you know? But yeah, e- ever since then, every time I walk, go out onto a conference talk stage still today, I ask, like, ask myself, I have this tiny doubt in the back of my head, like, it's never happened since but... Yeah, you just, I think you have to go with it with these things, you know? Like, when life kind of throws you these, whatever, curveballs, you have got to kind of adapt and it's not that big a deal. None of these things are that big a deal at the end of the day, you know? You kind of move on, live and learn. So yeah, but I still hope it doesn't happen again (laughs) .

   13. LRLenny Rachitsky11:02

       I also hate public speaking and I always fear this is exactly what's going to happen to me. And so, I think this is, uh, nice to hear that even when the worst possible thing basically happens, things can survive.

   14. PAPaul Adams11:13

       You can turn it around, yeah.

   15. LRLenny Rachitsky11:15

       A second area I wanted to hear from is your time at Google. And there's a couple products you worked on at Google, both of them were not- not what you'd call big successes. And then there was kind of a transition to Facebook, which was also kind of messy. Can you just share a couple stories from that time?

   16. PAPaul Adams11:31

       Yeah. Uh, similar to the Fa- to the kind of like, you know, walking off stage thing, um, you live and learn and, uh, Goo- I, I was at Google for four years and I was at Facebook for kind of two and a half years or so. At Google, I worked on a lot of failed social projects, like you mentioned. Google Buzz, um, Google, then later Google Plus. I think a- a lot of the motivation for those projects came from a place of fear. You know? It didn't come from a place of let's make a great product for people, let's, like, really understand the things people struggle with when communicating with family and friends, like, let's really, really try and create something wonderful. It came from a place of fear. And, uh, and so during those times, I kinda lear- I learned, I think, how-... how not to lead, uh, in places. And by the way, I should say, you know, at the time in Google, there was other things happening that were amazing. Like, Google were building Google Maps. A incredible product, one of my favorite products. I think one of the best products ever made. They built, were building Android, you know? It was kind of in, I was in the mobile team, in the mobile apps team at the time that Android came out. So like om- you know, incredibly good product. So, I just happened to be in the social side, which wasn't as (laughs) good. And, uh, yeah, we, we, um... Google Buzz was kind of a privacy disaster, and Google+ s- similar. And so, kind of halfway through, I, I'd kind of published research about groups, and how... I'd done this, I'd done a ton of research. A ka- an interesting kind of side note there is, at the time I was being asked, I was working in the research, in the UX team as a researcher, I was being asked to do a lot of tactical research, like usability study type stuff. Um, like can people use these products? And I ended up doing, um, a lot of formative research as well in the same session. So I sp- I'd kind of say to the team like, "Hey, I'll do the research and I'll answer your questions, but also I'm gonna do this other thing, and I'm gonna take 20 minutes doing that." And so what we used to do is, um, or what I used to do with people was map out their social network, all the people in it, their family, their friends, how they communicate. We'd map out all the channels, we'd talk about what worked well, what didn't. And ta- we did this with dozens and dozens of people over like the course of maybe 18 months. And the same pattern emerged every single time, which was people need way better ways to communicate with small groups of family and friends. And like, I, I kind of look back now and go like, "WhatsApp?" You know? Or like, maybe like iMessage if everyone's on Apple. But like, really obvious in hindsight, but at the time not obvious. And, uh, uh, and so we kind of tried to build a product around that called Google Plus, but, um, again, it was kind of motivated from the wrong, came from the wrong place. And so, halfway through the research that I'd p- that I'd kind of done, all this research had been made public through a conference talk, and, uh, Zuck and Facebook noticed, got in touch, one thing led to another, and I left and joined Facebook. Which was an amazing thing for me, personally. I, uh, Facebook was amazing. An amazing place at the time, and exciting, and, and they were trying to do things for the other reasons, the, the kind of good reasons like, "Hey, let's build an amazing product for people."

   17. LRLenny Rachitsky14:22

       And this was during Google Plus being built, you basically shifted?

   18. PAPaul Adams14:25

       Yeah, midway. It hadn't r- (laughs) I'm stressed that I'm even telling you about it. The project hadn't been launched, it was still under wraps, you know. Um, it was highly confidential. Google had done a lot of things at the time that, that, that were the first for them. I don't know if they've done them since, but things like everyone who worked on Google Plus was sent to a different building. That building had different, had a different keycard. If you didn't work on Google Plus, you could not get in. All sorts of like, kinda, uh, counter-cultural things at the time. And as a result, there was a lot of, you know, antagonism internally for Google Plus. And so when I left in the middle of the project, kind of leaving with all of the plans in my head to the enemy, you know. Some people saw me as a traitor, understandably. Uh, other people thought I was enlightened, you know. (laughs) It depends who you talked to. But it was, um... I was a, like, it was the right thing for me to do, but at the time, you know, it was, um, it was a hard thing to do.

   19. LRLenny Rachitsky15:17

       I know there was also like a lot of scrutiny in what you took with you, and the process.

   20. PAPaul Adams15:22

       Uh, yeah. When I left, uh, Google kinda assumed that I was one of the spies. You know. I was quarantined when I told them I was leaving, uh, they, uh, you know, forensically analyzed my laptop, like all sorts of stuff like that. So, it was pretty intense. Um, you know, looking back, I, I can understand why that happened. But the root cause for me is that the project has been run from a place of fear, competitive fear, which I don't think leads to good things.

   21. LRLenny Rachitsky15:53

       So one of the themes through the stories you just shared is, let's say failure, is, is... (laughs) I don't wanna make it that harsh, but just things not working out. And I'm curious as a product leader how important do you think that is for people to go through? If you think that's something that is almost a good thing, and I guess just is there anything there that you find helpful as a, as a coach, as a mentor, as someone, to people that are trying to become basically you?

   22. PAPaul Adams16:19

       Vary. Vary. Uh, still is, it still is, you know? Like, um, I have personally failed so many times. I, you know... Like, there are two stories, and the Google one is like long, deep, deep tentacles. But there are two st- there are two stories. I, I, I've failed a, a ton of times. Like at Intercom, I remember like, you know, I was like, when I was at Facebook I was very happy, and, um, Owen and De- I knew Owen and Des, who are the co-founders of Intercom, and, uh, they were trying to persuade me to join Intercom. We were like, it was like a 10 person company at the time. But Owen said something to, something to me at that time which, uh, has stuck with me ever since. He said, "You know, at, at Facebook you can g- you can design the product, but at Intercom you can design the company." And that was extremely appealing to me, like a great pitch. Uh, he's like, "Just design the company with us that you wanna work in." And so the, and so part of that was a company that embraces failure, that says it's okay to try things. I'm a big believer in like big bets. You know, higher risk, higher reward. Uh, I, I, I don't get as, as excited about incremental things. Now, having said that, there's of course a place for that too. Especially as companies get bigger. But I, I, I get excited about like bigger, big bets. And if you make big bets, you're gonna get th- a lot of it wrong. So a lot of the principles that we've built here at Intercom around building software, like we have a principle called Ship to Learn, and, uh, our sh- and we've actually changed it since. It's still there on the wall here. Uh, ship fast, ship early, ship often is what it says now. Used to say ship to learn. Ship fast, ship early, ship often. So like in that idea is the idea of failure. You know, you're gonna, y- it's not gonna go right, and, uh, it's gonna go wrong more often than not. But if you ship early and fast, and learn fast, you can change fast, and you can improve fast. And that's kind of how we, that's the kind of culture that, that we, as much as possible, try to embrace, uh, and teach people. But it's much easier said than done.

   23. LRLenny Rachitsky18:03

       (laughs) Yeah. Especially when you're in the moment, like, "Goddammit, everything's gonna fall apart. I really, I really messed this one up."

   24. PAPaul Adams18:10

       Yeah. And there's a trade-off w- with, with quality that people really struggle with. Like, you know, we, we've high standards of ourselves. A lot of Intercom comes from a kind of design founder background. We, we value the craft a lot. ... we never want to be embarrassed by what we ship. So there's a real tension there, a real trade-off, uh, where people have these high standards, which we encourage, and we encourage them to ship fast and learn and make mistakes. Uh, it's a constant kind of tension that we're navigating.

4. 18:38 – 33:19

   Tom Conrad: Lessons from Pets.com and Quibi—two of the most famous product disasters of all time

   1. PAPaul Adams18:38

   2. LRLenny Rachitsky18:38

      Next up we have Tom Konrad, who is Chief Product Officer at Quibi and engineering leader at Pets.com, two of the most memorable failures in product history, sharing his lessons from those wild experiences. Tom is currently CEO of Zero Longevity Science, which is a killer business and an app, in case you haven't come across it. I'd definitely check it out. Here's Tom. You brought up this phrase of notable disasters, and I want to talk about that. You worked at two of the most famous notable disasters of product companies, Pets.com and Quibi. I think it's really rare someone sees the inside of so much hype and then such a fall at a company. And so, I just want to spend some time in these two areas. And maybe the way to set it up is just what's a lesson you took away from each of these two experiences that you've taken with you to future work, and maybe advice you share with people?

   3. TKTom Konrad19:30

      Probably the biggest lesson is not really about, like, the, the, the specifics of the business. The biggest lesson really is these things make you better. They, in some instances, actually I think in both instances, they became kind of dominoes that, that opened doors f- for me in my own ambition and my own sort of, you know, professional life that maybe just wouldn't have opened at all if it, if I hadn't gone to those companies and learned those things and had those experiences. And, and frankly even in the case of Pets.com, like even the high profile nature of it, I could have worked at, I could have worked at, uh, you know, one of 1,000 e-commerce websites in 1999. And when I went on to some subsequent job interview or something and talked about my experience, they were like, "Oh, never heard of the thing that you worked on." But everybody certainly heard about Pets.com. It's a pretty funny example too of how some struggles are, like, timeless, you know, that, you know, that was 20, 23, 24 years ago now. And while as a leadership team we made I'm sure all kinds of mistakes, one of the things that happened was that there were three kind of over-funded pet e-commerce sites, and we all raised in excess of $50 million, um, which was, is a tremendous amount of money now, was a tremendous amount of money then. And we all thought it was a zero sum game in that we as one player started to spend on promotion or to spend irrationally on, you know, national broadcast television advertising, we all did, and it became this kind of unwinnable arms race. So there was like a, there is like a I think a fundamental lesson about the, like, the, you know, having an excess of, of investment can be its own albatross or, or, or, uh, lead you to take, make decisions that maybe would be or unwise. And then of course it's just, like, you know, timing is really important. You know, the, you know, Chewy is a online pet store, it's worth $9 billion today. They were a private company and bought by PetSmart and then spun back out. But when they were bought by PetSmart, they were acquired for $3 billion, you know, uh, biggest e-commerce acquisition of all time. And while I think it's probably unfair to compare Chewy who, who executed exceptionally well over a decade, grew their business brick by brick and turned it into something really remarkable, to Pets.com which was, you know, in a very, very different moment in time and, and, and tried to go to market in a really different way. The critique that is often leveled at Pets.com, or at least at the time, was like, "This is just a stupid business. They're shipping dog food around. You could never make that work." And like, that, that's just wrong. Like, you absolutely can make it work. Probably can't make it work when 80% of the country on the internet is still on dial-up. It was really, really early. (laughs)

   4. LRLenny Rachitsky22:25

      I saw a stat I think you shared somewhere that you took Pets.com from nothing to a public company to completely out of business in 19 months.

   5. TKTom Konrad22:33

      Yeah. Yeah, I think that's about right. You know, the, the, the other thing that's forgotten in the tale is that we, we actually didn't go bankrupt. We shut the company down and returned the, the remaining balance to the investors, which no public company had ever done before. And the leadership team just, like, reached the conclusion that, that given the way market conditions had evolved, there was just no way we were gonna be able to get more capital into the company, and, and it was a company that required, you know, additional investment to get to profitability. And, and so it was better to sort of wind down early, take the money that we, you know, we had in the bank and get it back to investors than to just spend every last penny on like a, you know, what was sort of a fruitless attempt to s- to salvage it.

   6. LRLenny Rachitsky23:21

      Hm, did not know that. Let's talk about Quibi. What went wrong there? Do you think there was a path to Quibi having worked out? Any big lessons that you took away from that experience that you bring with you?

   7. TKTom Konrad23:33

      The kind of miraculous thing about Quibi for me was it re-lit my enthusiasm for the industry for doing this work. I had left in I think it was December of 2018, and I thought that maybe I was just done making software. I had done it for a really long time, I'd done it for, like, 25 years or something, and I had changed a lot, the industry had changed a lot, and I thought maybe I just didn't have the same passion for it that I had, you know, a decade before. And it also seemed like maybe it would be fun to have, like, another chapter of my life that was just completely different, and I had a whole list of things that I thought I might want to do. I mean, they were really, they were kind of ridiculous. It's like, "Maybe I want to be a pastry chef. Maybe I want to be a landscape photographer. Maybe I wanna, you know, learn to make bad music to put up on SoundCloud or something." And the, the really only thing they had in common were they were all things that I knew nothing about. People would be like, "Oh, you- you think you might want to be a pastry chef? Do you like to bake?" And I'd be like, "No. I don't know anything about baking." You know, "Oh, you think you might like landscape photography. Do you take photos?" "No. I don't make photos." But I was kind of committed to the bit, actually to the point where when TechCrunch interviewed me about my departure from Snapchat, I was like, "You know, I'm out. I'm gonna do something else entirely." So that story is very much out there. But a few months, uh, after my last day at Snap, I got a call from Meg Whitman and Jeffrey Katzenberg who were starting up, it was called NewTE at the time, and you know, the pitch was, "We're gonna try to take the best of, um, mobile and Silicon Valley and consumer tech and sort of weld it to the best of sort of Hollywood style content production to build something like completely bespoken, purpose-made for consumption on the phone." They were looking for both technology leadership and product leadership and wanted to know if I was interested in one or both, and I took the meeting even though I wasn't really taking these kinds of calls from anybody, it just seemed like who's gonna pass up the opportunity to have lunch with the two of them? So I listened to the pitch and- and politely declined and told them that I was gonna be like a pastry chef or something. And we kept doing that every couple of months for like seven months. We'd go to lunch, they would give me an update on the progress they were making, and I would decline, you know, the invitation to get involved somehow. And then late in that year, I, uh, went to lunch one more time and Meg explained that they brought on someone to lead technology and they brought another person to lead product, and both of them for really truly for reasons that are completely disconnected from- from Quibi itself, both of them had left after about six weeks. And Meg's like, "We've raised all this money and we've told the world that we're shipping this product in about a year. We got an awful lot to do and I really could use some help and I would like consider it a personal favor if you were to come and spend just a couple days a week helping." You know, she's like, "I'll continue to look for someone who actually wants the job, but it would really be, it'd be, you know, great if you could help me get this off the ground." And, um, my wife is a freelance writer, marketing strategist, and loves her life as a freelance contributor and she's like, "You should do this. Like, why not? It's two days a week, it's just a few months, what's the worst thing that could happen? Like maybe- maybe you'll like it?" And- and I'm like, "No, no. The worst, here's the thing that will happen. Like, I won't do it two days a week. It will like immediately be three days, then four days, then five days, then six days. Like, I just know myself." And she's like, "No." She's like, "You know, just on Wednesday night at 6:00, close your Quibi laptop and be like, all they're paying me for is for Tuesday and Wednesday, and then open it back up on Tuesday morning. Like, that's all you've got to do." Well, you know, she's right about most things, but she's wrong about this. I fell, you know, deeply into it right away. And it was just so fun to get to build a team from scratch and to design and build a product from scratch and to take advantage of all of the sort of modern software architecture stuff that had come into being over the course of the 15 years since we had started Pandora. And, you know, I'm- I'm, you know, I'm- I'm embarrassed about- about some of- of what happened with Quibi for sure, but I'm super grateful for the experience because I just really fell in love with the industry again and was reminded of just how rewarding it can be to- to build something and- and to try to put it out there even if- even if you- you stumble pretty mightily along the way.

   8. LRLenny Rachitsky28:28

      Is there something that you took away from that experience that taught you what to try to avoid, what to try to pull towards?

   9. TKTom Konrad28:36

      I think I sort of m- m- misunderstood or misjudged companies sometimes by thinking about them like really focused on the- on the- the product execution. You know, kind of if you find an interesting problem that people, you know, people care about and you- you solve that problem in a really beautiful, elegant, delightful way that's 10 times better than- than anything else that they can get in that same space, they'll tell their friends and like all the rest will sort of take care of itself. And so that was always my ambition. Find a thing that I cared about building, do a great job building it a really delightful way, go really deep on listening to people and their feedback and iterate your way to success and sort of breaking through that membrane that we all strive to get across to, you know, kind of really great word of mouth. But I think the thing I've- I've come to better appreciate is that companies are also kind of like a, they're kind of a math problem that describes how you take, you know, investment and pour them into the equation and out the other side comes returns on some time horizon. And yes, there are variables in that equation that are influenced by the- the product that you build and all of the little details and decisions that you make about making that product great, but if the equation is fundamentally broken or a big swing in and of itself, no amount of like iteration and execution can like kind of get you out of the- the failed outputs of the broken equation. And I think, you know, Quibi made a bet that you could build an entirely bespoken content library that was sufficiently scaled to get people to subscribe and retain for a couple billion dollars. I mean, it was a huge amount of money. But, you know, we made, we made 70 shows in 18 months, which is, like, more content than all of the major broadcast networks combined made in a single year. So that was a pretty major accomplishment. And we made a bet that we would augment those sort of episodic and serialized for Hollywood style shows with a bunch of daily content that we produce at the level of, like, network television, nightly news and so forth, that would be an alternative to some of the sort of daily content that you might otherwise get on YouTube. And that was gonna be about a third of the content spend. One, like, super interesting thing that no one talks about is that all of that content was designed to be made, like, day of or day before it aired. So there was no back catalog of it, and it was all designed to be shot in these professional studios that we built out. And it was really expensive. Like I said, it was like a third of the, of the investment we were gonna make in content, maybe half, almost half the investment we were gonna make in content. And we launched two weeks into COVID, and we couldn't make any of that content except literally in the garages of the hosts' homes. And so we had this thing that was supposed to seem like really set apart from YouTube that literally now was being made exactly like YouTube content, which is sort of like self-produced at home with, you know, very little sort of, of the support infrastructure of Hollywood. Now, you can argue, I mean, I think the content on YouTube is really, really exceptional in this category, and maybe we were never gonna do better than that. But I think what was really fundamentally broken with Quibi was that the actual foundational equation of can you make enough premium content that's totally bespoke and made for the service and takes advantage of the unique nature of the phone, and is that enough content to get people to sign up and retain, and can you do that for a couple billion dollars? And I think the answer is no. The library has to be much, much bigger and you have to have, like any company, you have to have sufficient time and energy to iterate on the f- content format itself, 'cause we were really, our roadmap really wanted to innovate on the content format. And so I think part of what happened is pretty quickly it became clear that the math was just wrong. It wasn't gonna take two billion. It was gonna take six or eight or 10 billion. And the risk/reward profile of betting 10 billion on the format was just more than, than anyone could stomach.

5. 33:19 – 39:00

   Sri Batchu: When you fail, make sure you fail conclusively

   1. TKTom Konrad33:20

   2. LRLenny Rachitsky33:20

      Next up, we've got Sri Batchu, former head of growth at Ramp, who shares something that I've thought about ever since we had this conversation, which is this idea that when you fail, make sure you fail conclusively to make this failure an actual learning that you can build off of versus just a waste of time. Here's Sri.

   3. GSGuest (growth/experiments story)33:38

      Growth, uh, experiments in my history, uh, are typically, like, 30%-ish success rate. So the vast majority of things that you try don't work. And so y- you want to create a culture where people aren't afraid to take risks and, and aren't afraid to fail. And for me, failure is not that you didn't drive revenue. Failure is not learning. Like, so it's really important that you learn when you fail. Uh, and, uh, and so we, we celebrate failure as long as you're learning. And, and you can only learn if you've designed the right test and you failed conclusively. Because otherwise, I think many of us have been in situations where there's intuition that something might work and it doesn't work, and then you end up doing it over and over for years, like, 'cause every time a new executive or somebody else has the same idea, you try it again. And, uh, and it's because you haven't been able to design the, the test to, to fail conclusively. And it's, it's, it's hard to do, but at the end of the day, there's only two, two ways to, you know, make a, a, an experiment successful. Uh, either you have a very large end, uh, or you have a very significant treatment, which is, like, you know, what you're doing in the experiment itself. And, you know, in B2B you don't usually have the luxury of large end, which you do in consumer right now. Facebook can get stats to you in two hours. Uh, you know, a B2B company could take two years to, to get to the same number of, you know, touch points. And, uh, and so to counteract that, I, I recommend people, like, just trying to maximize the treatment effect, which is like if you have a hypothesis that you're testing, just throw all of the possible tactics and resources that you think would move that needle, 'cause you can always cost rationalize later if it works, right? Uh, and so just maximize the treatment effect and if, with all of that it didn't work, then you can say, "Hey, like, we're not gonna try this again, uh, because we literally did try everything that we could to test this hypothesis. And if it doesn't work in the best version and it's expensive as it is, this is not worth spending more time on." Uh, but if it does work, great. Then you, you know, do another version of the test with, like, half the tactics or whichever tactics you think work better or worse, uh, and you optimize over time.

   4. LRLenny Rachitsky35:49

      Is there an example you could share when you did that?

   5. GSGuest (growth/experiments story)35:52

      I mean, account-based marketing is something that is very common in, uh, in enterprise, uh, software, right, where you've selected, uh, you know, uh, certain customers that you think are high priority and you, you're saying, "I wanna touch them in as many, you know, nuanced ways possible, uh, to see if that drives conversion." And this is something, you know, I've seen tried many times, where people do it, but they kind of, you know, uh, do it halfway, where they're like, "Okay, we tried these three things. Conversion of the control group, like, wasn't higher, uh, and so we're, you know, we think it's just not gonna work." And then a new go-to-market executive comes and they have to do it again, they have to do it again, they have to do it again. It's, like, a very common one wherever this happens. And, uh, and so when we did it at Ramp, uh, we, we did exactly, you know, what I just described, which is, like, let's really be thoughtful about the experiment design, both in terms of, like, maximizing the number of people as well as, you know, maximizing the number of ways and types of ways that we're effectively, uh, touching these target customers to, to show the value.

   6. LRLenny Rachitsky37:03

      So what it sounds like is the hypothesis isn't like, this email will have a big impact on conversion. It's like this strategy of coming after customers is what we're testing?

   7. GSGuest (growth/experiments story)37:15

      That's the example there, right? Like, and I think like if you, for example, if you have the, it- it- it's a, this kind of framework is more important for cross-functional, larger scale, bigger tests, uh, rather than like, you know, an email modification, right? But you can even use it on a micro example, like an email modification, right., where you are like, "Okay, I think this particular email is underperforming because it's not talking to this part of the customer's pain point or journey," or what have you. And, uh, and you could just, the simplest test would be, "Okay, let me make some tweaks and to the text and edit that." And that could be the end of that test. And if that doesn't work, you're like, "Oh, maybe those weren't the right text edits. Let me do different text edits." Or whatever. And that's fine. That's low cost. It's not the end of the world and it's, uh, for you to be wrong there. But an alternative that you could do is like, "Oh, what are all of the things that I could change about this email, uh, in the same test?" Is it the trigger of the email? Is it the text content of the email? Is it, uh, you know, additional personalization? Is it the design of the email? Like, trying to think about what are all of the various levers that, uh, you think could be wrong and put them all together to test your hypothesis of this touchpoint is wrong, uh, and how do I improve that?

   8. LRLenny Rachitsky38:34

      Well, obviously the downside of that is you don't, if it doesn't work, you don't know if it's like, oh, maybe it was this thing could have worked and the subject wasn't right.

   9. GSGuest (growth/experiments story)38:40

      Yeah, yeah, yeah. So there's, there's always trade-offs on this. And so, but what you're hoping is like you've done a, a complete refresh where you did all the things that you thought were intuitive that should work, and if it doesn't work, then you're like, "Okay, maybe my hypothesis is wrong." But you're right. There's always gonna be a challenge if maybe the execution is wrong, uh, and I did too many things, uh, potentially in that

6. 39:00 – 44:32

   Jiaona Zhang (JZ): One of the biggest product misses at Airbnb

   1. GSGuest (growth/experiments story)39:00

      case.

   2. LRLenny Rachitsky39:01

      Our next story is from JZ who is a colleague of mine at Airbnb, head of product at Webflow when we recorded this episode. And this is her sharing the story of one of the biggest product misses at Airbnb. You've seen a lot of new PMs and you've seen these PMs succeed. You've seen some fail. What are the most common mistakes that you find new PMs make in this experience of kind of helping new PMs get into the field?

   3. GSGuest (Airbnb Plus failure story)39:26

      I think something that is really hard to untrain, but I think every human does it, is you jump to solutions. And so one of the biggest things I see not just in my course, but also just as a PM and like some of the mistakes that you make as a PM, is the idea of you get really attached to a solution, a way of implementing something, something that you, you, and you can see in your head that you want to build. And so that's the first thing I really want to like unteach in, in our course. And so a lot of people will literally come in, they'll be like, "I want to build X startup," or, "I want to do, you know, this thing," or, "I, I'm in blank school and I've been doing a lot of research on this particular area." And so like untraining that and being like, "Hey, we're going to go out there. We are not gonna think at all about the thing that you want to build, but instead we're going to be focused on users and pe- like people in the real world and their problems." And the first step is to understand their problems and then to understand if there's an opportunity here as opposed to, hey, you want to build X thing for Y person. So that's the biggest mistake that you really have to unteach and like retrain thinking around.

   4. LRLenny Rachitsky40:25

      So let's kind of go to the other side of this question. We talked about what mistakes new PMs make. I'm curious, what's the biggest product mistake that you've made?

   5. GSGuest (Airbnb Plus failure story)40:34

      Wow, that's a good one. It's so interesting. I feel like as product people, we're always making mistakes and we're always learning. Maybe I'll, I'll give an example from Airbnb since you and I were both there.

   6. LRLenny Rachitsky40:43

      Perfect.

   7. GSGuest (Airbnb Plus failure story)40:43

      And, and this one does stand out to me. So, you know, we were working on this concept called Airbnb Plus. You know, if you took a step back, what we were really trying to do is to be like, hey, not everyone trusts Airbnb in terms of, you know, it's- it's- it's a platform. It's not like it's managed inventory. It's not a hotel. How do you go in and really make sure that we're, like all the Airbnbs are meeting the quality bar? But I do think we were very solution first, and I think we were also competitor afraid at the time, right? So it was during a time where there were managed marketplaces, there were the- the Saunders out there. And I think that as a, as a company, we're very much like, oh my goodness, like what's, what are we gonna do in the world of managed marketplaces? And so we went really hard down the solution space. We essentially were like, "Let's go inspect our inventory. Let's actually try to manage our inventory more." And really what we should have done is taken a step back and be like, "What's the real problem?" The real problem is people want to know what they're getting themselves into. We need to represent the homes a lot better. And I think the other piece here that's really important is what, as a company is your strategic strength and like what's in your wheelhouse? So for example, Airbnb, we weren't that strong in operations. We again, we're this platform with this marketplace, right? And so if you don't have that muscle and then you're asking the company, the teams to essentially build it from the ground up, that's really, really difficult. Not to mention the unit economics. Is- is, are the unit economics actually going to work even as you scale?

   8. LRLenny Rachitsky42:03

      Yeah. I feel like Airbnb Plus is an untold story that somebody should tell. And that could be its own podcast, I guess.

   9. GSGuest (Airbnb Plus failure story)42:09

      You and I can tell it. (laughs)

   10. LRLenny Rachitsky42:10

       We could tell, this could be Airbnb Plus the hidden, the story. As you said, the problem it was trying to solve was people don't really trust, they don't wanna even consider Airbnb 'cause it's like, "No, I don't want to stay in someone's home. I don't know what it'll be. It's unpredictable." And so as an outsider, it felt like a really clever approach. We're gonna vet them, we're gonna make sure they're awesome. There's a minimum bar. I think the, and I guess this is the question is do you think it was just like this is never possible because there will, we'll never make money as a business doing this 'cause we don't make that much per booking and investing time, resources, sending people pillows, all that stuff is ever gonna be economical? Or do you think there was a path and it was just not executed well?

   11. GSGuest (Airbnb Plus failure story)42:51

       I- I think there wasn't really a, a clear path. I, I think there was lots about execution-

   12. LRLenny Rachitsky42:55

       Yeah. That's my impression.

   13. GSGuest (Airbnb Plus failure story)42:55

       ... exactly. And it's more just like if you understood, again, this is what my point around unit economics. There are things where I think you have like magical thinking around unit economics. You're like, "Well, when we get to the scale of X, we-"It's all gonna work out. We can make these things happen. I think you actually need to really make sure the unit economics work right at the beginning. So that is definitely one lesson. And I think the other thing is, aga- and going back to the spirit of what are you trying to achieve? If you're, if you're trying to achieve this idea of, like, really knowing the quality of the place, and, and for a platform like Airbnb, the right way to go about doing it is through our reviews, right? Through our guest reviews, which are essentially free, as opposed to literally sending out inspectors. And I think the, the other things are, if you can get signal on what are the things around quality that people care about? Is it cleaner, uh, cleaning? Is it, is it the, uh, "Hey, I'm, I'm locked out." And I think that there are other solutions besides inspection that then get at that. So for example, it is actually cheaper to go send everyone a lockbox than to deploy an inspector (laughs) and go look at your property, right? It is actually cheaper to maybe do a partnership with, like, a, you know, a bunch of cleaners in different local areas and then get that as part of the fee, like, as opposed to doing the inspection. So again, it's really about what are you really trying to achieve? What is the user problem in each of these areas? And can you target that problem with the particular listing that is, you know, that you're looking at? And so I, yeah, I, I personally don't believe the unit economics ever would've really worked out. I think we should've known that... Or, like, we should have, uh, dug into that more at the very beginning and then to get very tailored instead of, like, one blunt instrument to solve it all, "Hey, we're gonna go inspect." It's like, what is the problem for this listing and what's the best solution to fix that problem?

7. 44:32 – 57:57

   Gina Gotthilf: Everyone has an “A side” and a “B side,” and we should all share our B sides more

   1. GSGuest (Airbnb Plus failure story)44:32

   2. LRLenny Rachitsky44:33

      Our second-to-last story is from Gina Gotthilf, who was an early growth leader at Duolingo. She's currently COO of Latitude, and this is her sharing a wide-ranging and important point about how everyone has both an A side and a B side to their career, and people often only share their A side. So, this is Gina sharing her B side.

   3. GSGuest (Duolingo failures/B-sides story)44:53

      We are very encouraged in our lives, especially professionally, to talk about our A sides all the time, because that's what impresses people, that, that's what opens doors, that's what allows us to keep growing, and it's so important. So, it means that a lot of what you hear in podcasts and on, on stage ends up being the Instagrammable version of someone or a company or a country's trajectory. It's just the highlights. You know, a- and when I talk about my A side, it's very impressive. I, you know, I did things like, we'll talk about, I met President Obama. I worked on the Mike Bloomberg presi- Mike Bloomberg presidential campaign. I helped Duolingo scale from three to 200 million users. I worked with Tumblr, helping them scale in Latin America. Andreessen Horowitz invested in my company, et cetera. But between all of those highlights, there were so many B moments that get sort of, like, shoved under the rug because it's just easier for me and it's more impressive for others. But I really like to highlight those, because I think that most of us have a lot of B moments every day, every week, every month, and every period of our lives, and it's easy to think that things aren't just not gonna work out for us because we're in one of those B moments if we don't recognize them as moments.

   4. LRLenny Rachitsky45:59

      I love this concept. We're gonna talk, as you expected, about a lot of your A side stuff.

   5. GSGuest (Duolingo failures/B-sides story)46:04

      (laughs)

   6. LRLenny Rachitsky46:04

      Is there any example of a B side story of your life that would be interesting to share?

   7. GSGuest (Duolingo failures/B-sides story)46:09

      Look, I think those are the most interesting, because they're, they're, they're funny or ridiculous, you know? I had a lot of B sides, and I still do. For example, I, you know, I had no- I had no idea what I wanted to do. I actually wanted to be an... I thought I wanted to be an actress. I either wanted to be that person in-

   8. LRLenny Rachitsky46:25

      Wow.

   9. GSGuest (Duolingo failures/B-sides story)46:25

      ... SeaWorld who goes like this with the dolphin. This was before SeaWorld was canceled.

   10. LRLenny Rachitsky46:29

       (laughs) Yeah.

   11. GSGuest (Duolingo failures/B-sides story)46:29

       Or I wanted to be an actress. I applied to schools. I didn't get into any Ivy League. I didn't get into any of the top schools I wanted to go to. When I got to college, I actually ended up dropping out because I got so depressed, like, i- incredibly depressed, couldn't-get-out-of-bed depressed. Ironically, I dropped out of Reed College, which is the same college that Steve Jobs dropped out of. So, you know, I was just destined for greatness. I knew it at that moment. (laughs)

   12. LRLenny Rachitsky46:53

       It all makes sense looking backwards, as you said.

   13. GSGuest (Duolingo failures/B-sides story)46:55

       Totally. I was dropping out being like, "Yes, this is exactly the path." No, I was, I was miserable. I thought there was no path forward. And when I... And I finally went back and graduated. The co- The college counselor looked at my curriculum and said, "What have you even done with your life? There's nothing to show for." And it was shocking, because I was always, like, the overachiever who wanted to do the maximum curriculum and, like, ace all of my classes and do whatever, you know? Like, I, I did three diplomas in high school, the international, the American, the Brazilian. Um, and so I real-... That, for me, I think my one learning there that has stuck with me, and I think can, can work for other people too, is that it's not just about doing things that actually matter and, like, l- and, and learning. It's about being able to tell a story, and it's about understanding what other people perceive as valuable. I applied to a hundred companies. I didn't hear back from most of them. I finally got an internship at kind of, like, a tier B/C digital marketing agency in New York City, because I wanted to live in New York so badly. Um, and they forgot to apply for my visa on time, so I lost my visa and had to go back to Brazil. And then I ended up leaving that organization to go work for another one, and, like, I won't even go into the details of the, the shadiness of that company that I worked for. But then they ended up laying me off, so I lost my visa again, had to go back home. Found another opportunity, got fired that time. So, there was just a lot of rockiness in my start that I don't think you would imagine when you see someone up on stage, like, leading a conference for 5,000 people, um, that I think is important. And even when I started working for Tumblr, I was like, "This is it." Like, "I made it. This is a really interesting company. This is gonna work out." That was super rocky, because it was an early-stage startup. So for example, you know, they couldn't figure out how to wire money to Brazil, so I was not paid for six months, and at one point, me and, like, my colleagues were trying to get money out of the teller to pay contractors because we had no money to pay them, and, like, we borrowed money from people, and, and finally they also laid me off, because they decided to sell to Yahoo!. And then I had to figure out, like, "What am I gonna do? No one's gonna hire me." I've been fired and laid off so many times. So, this is all before I started, like, an agency to help US-based tech companies and startups grow in Latin America, because I figured I was in, like, this really great place to make that happen. And it eventually worked for well-known companies such as Duolingo. At the time, they weren't well-known. They were a tiny little startup. They didn't have an Android app. And that's how I started working with Duolingo, because...... their head of marketing connected with someone they had worked with at Flickr and said, "I noticed Tumblr grew a lot in Brazil last year. Can you recommend, like, a company or an agency to help?" And they said, "This girl," and I was 26. And so they conn- that's how they connected me with Duolingo, and I started helping them grow in Brazil as a consultant. They were like, "This is great. Can you help us grow in Chile, Argentina," and I was like, "Yes." They were like, "How about Mexico?" And I was like, "Yes." You know, I- I... Did I know anything about these places, Lenny? Did I know people there? No. But like, you can figure it out. And then they ended up asking me to come on full-time, do that across the world, Japan, China, Korea, Turkey, Spain, France, et cetera. And then to own growth, which ended up meaning communications, social media, government partnerships, anything to grow, and then a lot of... And then eventually became an AB testing growth engine with engineers and PMs and designers, of which I knew nothing about. And you know, even after that, like, I- I left Duolingo five years later, didn't know what to do with my life. You'd think, "Oh, wow, you have it figured out. Now you left Duolingo, you have the world in front of you." And I'm like, "Maybe I can finally go work for nonprofits," which is what I actually wanted to do in the first place. Tried a hand at that, like had a couple of experiences before going to work for the Mike Bloomberg campaign. Working for the Mike Bloomberg ca- campaign is impressive, but you know what? Mike Bloomberg didn't win. He's not the president. So that was not a successful campaign, you know?

   14. LRLenny Rachitsky50:43

       Mm-hmm.

   15. GSGuest (Duolingo failures/B-sides story)50:44

       If you really look at it, um... And, and, and yeah, Latitude seems like it's a really promising path, but it, there's A days and B days, um, so i- it's just a lot of that and just staying resilient and s- and believing in yourself and getting back on the horse when you fall on your face.

   16. LRLenny Rachitsky51:00

       Amazing, that's such an important message. I think one of the threads from what you're describing, something that I think about a lot is, people kind of underestimate how long their career is. There's just so much time to do stuff and for things to start to work. This is gonna sound really fancy but I think Marcus Aurelius has this quote about how our life is actually very long, we just use it really badly, and we just waste a lot of our time.

   17. GSGuest (Duolingo failures/B-sides story)51:21

       I think you're so right, Lenny, and I love that because people are going around being like, "Life is short, life is short," but that's so true, we waste so much time. But also I think we don't, we don't recognize how much opportunity we have in front of us, and as a 26-year-old, I definitely thought my career was over. You know, I was like, "I blew it." And like looking back, it's funny, you know?

   18. LRLenny Rachitsky51:40

       Yeah. I know exactly what you mean. Like, I spent nine years at my first job at a random company in San Diego in a startup, and I was like, "What am I doing here so long?" (laughs) And then, and it turned out that was really useful for the thing I did next. And then eventually, wow, things started to really take off. So I think that's a really good lesson for people, just a long time. Like, this is my fourth career. Like, I've switched careers many times. I was a engineer, then I was a founder, then I was a product manager, and I'm whatever this is (laughs) , whatever you call this thing.

   19. GSGuest (Duolingo failures/B-sides story)52:06

       I guess me too. I was, I was an operator, I was a consultant. Well, I was an employee, I was a consultant, then I was an operator, which is a fancy way to say employee at a startup, and then now I'm a founder and a VC and an an- angel and whatever this is.

   20. LRLenny Rachitsky52:22

       (laughs) Awesome, awesome.

   21. GSGuest (Duolingo failures/B-sides story)52:23

       (laughs)

   22. LRLenny Rachitsky52:23

       So I think, I think that's a really important takeaway, is just there's a lot of time to do stuff and don't stress if things aren't moving as fast as you want. I'm curious, what's like a mess up or a big mistake maybe that you made or your team made that was like, "Oh wow, that was a big waste of time"?

   23. GSGuest (Duolingo failures/B-sides story)52:38

       Yeah, look, a lot of things didn't work out. More than 50% of our, of our AB tests didn't work out. We made bets that didn't make sense. I will say though that like in the spirit of A and B sides, I, and I think in general, we are really good at forgetting the B stuff. Like I, I talk so much about all the stuff that worked that it's hard to remember like all of those moments that didn't actually work, and the, and the moment and the thing that I, I tend to talk about, which is like this mistake that we made as the growth team is almost like one of those, like when you get asked in an interview like, "What's your, what's like your biggest weakness?" And you're like, "I'm a perfectionist." You know? It's like one of those things that actually makes you sound good because it's the story about how my team really wanted to implement badges. We spent a lot of time playing all the games that were popular at the time, trying to understand how those gamification growth hacks that we could find in, in those apps would potentially overlay onto Duolingo and how, you know, we would do that, and badges was just pervasive in all of the top games, and so it seemed like a, a no-brainer. But since we ranked all of our experiments in terms of ROI and return being like how many users we think we're gonna get from this, the AU's and inv- time investments, it was never... It never made sense to focus on this because we thought that the time sink would be too high. So I actually ended up not letting the team run this experiment for like six months so that we focused on lower hanging fruit, so that's like a mistake on my end. Then we decided to, we decided to run this experiment in the most lean way possible. We're like, "You know what? Like there's MVPS, there's like minimum viable experiments. We don't have to run a whole badges thing, we can just like do something more simple and actually see if that like leads to growth in an interesting way and then we'll know." And we ran this very simple experiment that was like, you sign up and then you get a badge, and it was like this girl with a balloon. I don't know, she was like happy or whatever, and of course, of course in retrospect it led to no results because no one is proud of signing up. It's not an exciting moment and you don't even have badges to collect, you can't show it to other people. Like, none of the things that make badges compelling were there. But we were like, "Okay, well, we tested it, it didn't work," and then we moved on. So we moved on for like another, I don't know, eight months and we didn't look back. And then when we did look back... First of all, at that point, we, we discovered that we hadn't been dogfooding, okay, which also was embarrassing looking back. We hadn't been dogfooding in the growth team. We'd just come up with hypotheses, we'd like... We were super careful about prioritizing them and making sure that we were doing the best possible like write-ups and all these things, but like the dogfooding piece, it just... I didn't come from a p- a product background. I was a marketer and I hadn't even... I didn't really understand the term dogfooding, but when I... When we thought a- we had a conversation, we were like, "You know what? If we had just tested that, we would have all known that this was a super lame badge." And I was like, "Why are we not testing our experiments?" And so like that became part of our, of our practice. It's still relevant. I, I just had a conversation yesterday with engineers at Latitude-I, I, I haven't explained what we built yet, where we're building it. Maybe we'll get there. But I was talking ............................ at Latitude and they're awesome. Like, in terms of our product team, we have, like, the number eight employee at Nubank. You might have heard of Nubank, but it's this, like, massive banking, like, fintech in Latin America. And, and we have people from, like, other fintechs and then we have this guy who's a lead PM at Twilio. And I was explaining to them why we should be dogfooding and they were all like, "Oh, yeah. We should dogfood." It's just, like, easy to forget stuff like that. So, so that was a mistake that, you know, we could have probably gotten to the growth that we got to with badges much earlier, earlier on. And not only did we get to growth with badges, but it became like this amazing treasure trove of opportunity because once you have badges and people want them, you can now ask people to do anything. Like, go find friends, go buy things, like, you know? Whatever it is. And so like we, we had, uh, we impacted almost all metrics across the company positively, espe- inc- incl-including some we hadn't expected. But it's, like, easy to talk about a mistake that ended up being a win, so that's why I s- I compared it to the interview thing in the beginning. But you know, we, we tried making Duolingo a social app really early on and failed. Um, it was called Duo- Duels. Duo Duels, like you could duel. You know?

   24. LRLenny Rachitsky56:42

       Very clever.

   25. GSGuest (Duolingo failures/B-sides story)56:43

       Yeah, I know. We were, we were clever, but people didn't use it and we didn't figure out why. We, we tried making a Duolingo for schools platform and like, we couldn't get it to pick up. I went and launched Duolingo in China, and it got download- downloaded by a million people in the first day and then the app got blocked. And I, we, you know, because of the government, and then we couldn't figure out what to do and then everyone rated the app, like, a one star because it didn't work. And so then we had-

   26. LRLenny Rachitsky57:08

       Mm-hmm.

   27. GSGuest (Duolingo failures/B-sides story)57:08

       ... a really, like, a lot of trouble actually recovering from that. We launched Duolingo in India and didn't realize because we couldn't have unless we went there, which we finally did, that most people set their phones, phone UI in India to English because typing in Hindi is hard and of course there's a lot of languages throughout India. And we were making it so that when you downloaded Duolingo, whatever L- UI you open your app, your, your phone was set to, we offered not that language for you to learn. That was your base language. So we were telling people, "Learn French, Spanish, German from English." And they were all trying to learn English, so they didn't find what they were looking for and they left. Um, there were so many mistakes, you know? And luckily, I think we were able to, to bounce back from most of them in terms of how Duolingo's doing today.

   28. LRLenny Rachitsky57:53

       And our final story is from Maggie Crowley, VP of product at

8. 57:57 – 1:00:33

   Maggie Crowley: Her favorite interview question about failure, and lessons from a personal failure

   1. LRLenny Rachitsky57:57

      Toast, and one of the most beloved episodes of the podcast. This is Maggie sharing something a little bit different. Her favorite interview question about failure and what it tells you about the person you're interviewing, plus a story of her own product failure. Here's Maggie.

   2. MCMaggie Crowley58:12

      A question I ask in every product interview is, uh, "What's the worst product you've ever shipped?" And that's because I, I don't think you're a good PM if you haven't shipped something that's really shitty. Like, you just haven't had enough reps, you haven't done it enough time, like ... And it, and it's not only that you've done it, but that you can admit it and you know which one it is. Like, that's so important. And I remember one (laughs) this is so dumb, I can't ... I'm still so mad about this, that we did this. It was, I won't name which team, which company, I'm not gonna call that out, but we wanted, we decided we needed to do a rewrite, red flag number one, of a existing product and, you know, an engineer who I'd worked with many times, we had a really good relationship, and this person was like, "Yeah, yeah, yeah, it's gonna take six months. Like, no problem." Core part of the product, been around for forever. Like, one of those things that the code is still, is still the code written by the founders kind of thing. It didn't take six months, it took two and a half years. It still wasn't done. It almost never ... Like, it went on for so much longer than it should have. It took us forever to get to feature parity. It was the worst project. Like, so many people rotated in and out of it, everyone thought it was dumb. Like, sunk cost fallacy. Just the worst. And it's because A, we got arrogant and we thought we could do it, B, we skipped discovery. We didn't really write a one-pager, we just went for it. We didn't do enough technical and design research into what the requirements would actually have to be. And there you have it.

   3. LRLenny Rachitsky59:45

      And it did not work out. Or was it a huge success in the end and it changed the trajectory of the business?

   4. MCMaggie Crowley59:50

      Absolutely not. But you know what? I didn't get fired, so like, it's fine.

   5. LRLenny Rachitsky59:54

      I feel like I've gone through those experiences and then like, three or four years later it's like another, "Maybe this rewrite and redesign may work, 'cause we've just, we haven't updated this thing in a long time."

   6. MCMaggie Crowley1:00:04

      Just don't do it. Don't rewrite. If anyone ever tells you to do a rewrite, don't do it. A side by side rewrite, nope.

   7. LRLenny Rachitsky1:00:11

      Yep. I've never had a ... Like, what I run into is once you get too far down a redesign/rewrite, everything went building in that new world and then you launch it and the experiment's negative and then it's just like, "Oh, we just got to launch it. We're gonna claw it back. We're gonna, we're gonna figure out how to get back to neutral someday."

   8. MCMaggie Crowley1:00:25

      Yeah.

   9. LRLenny Rachitsky1:00:26

      Yeah.

   10. MCMaggie Crowley1:00:26

       Yeah. Don't do that.

   11. LRLenny Rachitsky1:00:28

       Good times. And that is a wrap. I hope you enjoy these stories

9. 1:00:33 – 1:01:11

   Thanks for listening

   1. LRLenny Rachitsky1:00:33

      of failure. And I want to give a huge special thank you to all of our amazing guests for being vulnerable and sharing these stories of failure in their career. I hope you leave this episode with a new perspective on how setbacks and challenges and failure can often be exactly what you need to get to the next step of your career or your life. If you've got a great story to tell about failure, I'd love to hear it. Leave a comment either on YouTube or on lennysnewsletter.com or just DM me on Twitter or you could reach out at lennyrajitsky.com and click the big contact button. Thank you for listening. Bye, everyone.

Episode duration: 1:01:11