The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

I'm very clear that I'm a big fan of test everything, which is any code change that you make, any feature that you introduce has to be in some experiment because, again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising unexpected impact. And so I don't think it's possible to experiment too much. You have to allocate some times to these high-risk, high-reward ideas. We're gonna try something that's most likely to fail, but if it does win, it's gonna h- be a home run. And you have to be ready to understand and agree that most will fail. And I've, it's amazing how many times I've seen people come up with new designs or a radical new idea and they believe in it, and that's okay. I'm just cautioning them all the time to say, "If you go for something big, try it out, but be ready to fail 80% of the time."

(instrumental music) Welcome to Lenny's Podcast, where I interview world-class product leaders and growth experts to learn from their hard-won experiences building and growing today's most successful products. Today my guest is Rony Kohavi. Rony is seen by many as the world expert on A/B testing and experimentation. Most recently, he was VP and technical fellow of Relevance at Airbnb, where he led their search experience team. Prior to that, he was corporate vice president at Microsoft, where he led Microsoft's experimentation platform team. Before that, he was director of data mining and personalization at Amazon. He's currently a full-time advisor and instructor. He's also the author of the go-to book on experimentation called Trustworthy: Online Controlled Experiments. And in our show notes, you'll find a code to get a discount on taking his live cohort-based course on Maven. In our conversation, we get super tactical about A/B testing. Rony shares his advice for when you should start considering running experiments at your company, how to change your company's culture to be more experiment-driven, what are signs your experiments are potentially invalid, why trust is the most important element of a successful experiment culture and platform, how to get started if you want to start running experiments at your company. He also explains what actually is a p-value and something called Twyman's law, plus some hot takes about Airbnb and experiments in general. This episode is for anyone who is interested in either creating an experiment-driven culture at their company or just fine-tuning one that already exists. Enjoy this episode with Rony Kohavi after a short word from our sponsors. This episode is brought to you by Mixpanel. Get deep insights into what your users are doing at every stage of the funnel at a fair price that scales as you grow. Mixpanel gives you quick answers about your users from awareness to acquisition through retention. And by capturing website activity, ad data, and multi-touch attribution right in Mixpanel, you can improve every aspect of the full user funnel. Powered by first-party behavioral data instead of third-party cookies, Mixpanel is built to be more powerful and easier to use than Google Analytics. Explore plans for teams of every size and see what Mixpanel can do for you at mixpanel.com/friends/lenny. And while you're at it, they're also hiring, so check it out at mixpanel.com/friends/lenny. This episode is brought to you by Round. Round is the private network built by tech leaders for tech leaders. Round combines the best of coaching, learning, and authentic relationships to help you identify where you want to go and accelerate your path to get there, which is why their wait list tops thousands of tech execs. Round is on a mission to shape the future of technology and its impact on society. Leading in tech is uniquely challenging, and doing it well is easiest when surrounded by leaders who understand your day-to-day experiences. When we're meeting and building relationships with the right people, we're more likely to learn, find new opportunities, be dynamic in our thinking, and achieve our goals. Building and managing your network doesn't have to feel like networking. Join Round to surround yourself with leaders from tech's most innovative companies. Build relationships, be inspired, take action. Visit round.tech/apply and use promo code LENNY to skip the wait list. That's round.tech/apply. Rony, welcome to the podcast.