Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor)

1. 0:00 – 4:31

   Ramesh’s background

   1. RJRamesh Johari0:00

      Marketplaces are a little bit like a game of Whac-A-Mole. It... Like, one example that I came across, eh, with one of the companies I worked with that, that I love is our new, you know, supply side was having a pretty bad experience. So what we decided to do is build some custom bespoke features that were really gonna direct them to more experienced folks on the other side of the market. Good. And then yeah, lo and behold, you know, pretty soon those metrics start to look better. But then we're looking at it, we're like, "Wait a second. Now, you know, the existing folks on the other side are having a worse experience." So you kind of whiplash around. You're like, "Oh, wait a second, we better do something about that." So we take them, we try to match them up with the more experienced folks. And now suddenly, a month after that you're like, you know, "Wait a second," and, and your metrics just keep moving around. And that's because the Whac-A-Mole game here is ultimately a lot of marketplace management is moving attention and inventory around. Many of the changes that are most consequential create winners and losers, and rolling with those changes is about recognizing whether the winners you've created are more important to your business than the losers you've created in the process.

   2. LRLenny Rachitsky0:58

      (instrumental music) Today my guest is Ramesh Johari. Ramesh is a professor at Stanford University, where he does research on and teaches data science methods and practices, with a specific focus on the design and operation of online marketplaces. He's advised and worked with some of the biggest marketplaces in the world, including Airbnb, Uber, Stripe, Bumble, Stitch Fix, Upwork, and many others. And in our conversation we get super nerdy on how to build a thriving marketplace, including where to focus your resources to fuel the marketplace flywheel of growth, why data and data science is so central to building a successful marketplace, how to design a better review system, why as a founder you shouldn't think of yourself as a marketplace founder but instead simply as a founder, also how AI is gonna impact data science and marketplaces and experimentation, and so much more. If you're building a marketplace business or thinking about building a marketplace or just curious, this episode is for you. With that, I bring you Ramesh Johari after a short word from our sponsors. This episode is brought to you by Sanity. Your website is the heart of your growth engine. For that engine to drive big results, you need to be able to move super fast, ship new content, experiment, learn, and iterate. But most content management systems just aren't built for this. Your content teams wrestle with rigid interfaces as they build new pages. You spend endless time copying and pasting across pages, and recreating content for other channels and applications. And their ideas for new experiments are squashed when developers can't build them within the constraints of outdated tech. Forward-thinking companies like Figma, Amplitude, Loom, Riot Games, Linear, and more use Sanity to build content growth engines that scale, drive innovation, and accelerate customer acquisition. With Sanity, your team can dream bigger and move faster. As the most powerful headless CMS on the market, you can tailor editorial workflows to match your business, reuse content seamlessly across any page or channel, and bring your ideas to market without developer friction. Sanity makes life better for your whole team. It's fast for developers to build with, intuitive for content managers, and it integrates seamlessly with the rest of your tech stack. Get started with Sanity's generous free plan, and as a Lenny's Podcast listener you can get a boosted plan with double the monthly usage. Head over to sanity.io/lenny to get started for free. That's sanity.io/lenny. This episode is brought to you by Hex. If you're a data person, you probably have to jump between different tools to run queries, build visualizations, write Python, and send around a lot of screenshots and CSV files. Hex brings everything together. Its powerful notebook UI lets you analyze data in SQL, Python, or no-code, in any combination, and work together with live multiplayer and version control. And now Hex's AI tools can generate queries and code, create visualizations, and even kickstart a whole analysis for you all from natural language prompts. It's like having an analytics copilot built right into where you're already doing your work. Then when you're ready to share, you can use Hex's drag-and-drop app builder to configure beautiful reports or dashboards that anyone can use. Join the hundreds of data teams like Notion, AllTrails, Loom, Mixpanel, and Algolia using Hex every day to make their work more impactful. Sign up today at hex.tech/lenny to get a 60-day free trial of the Hex team plan. That's hex.tech/lenny. (instrumental music)

2. 4:31 – 8:10

   A brief overview of what a marketplace is

   1. LRLenny Rachitsky4:31

      Ramesh, thank you so much for being here. Welcome to the podcast.

   2. RJRamesh Johari4:35

      Thanks so much for having me, Lenny. It's, uh, it's great to be on.

   3. LRLenny Rachitsky4:38

      It's great to have you on. A big thank you to Riley Newman for connecting us. Riley was the first data scientist at Airbnb, and head of data science at Airbnb, and that role is actually a really good microcosm of what we're gonna be focusing on in our chat today. We're gonna get super nerdy about marketplaces and experimentation and data. I know that's your jam. Are you ready to dive in?

   4. RJRamesh Johari5:01

      I really am, yeah. And, and I actually wanna thank Riley too. Um, you know, I, I got to know Riley when I was at, at oDesk, uh, you know, first as a research scientist, and then I directed their data science team. This was like way back in, in 2012. And you know, I was looking around for people who are experts on how we think about data and marketplaces, and, and Riley Newman came up. And so I invited him to come talk to us at, at oDesk, and we've stayed in touch since then. You know, those were early days of where this industry was, and, and I've had a kind of lengthy career now, uh, thinking about those kinds of problems. So I'm pretty excited to talk about it with you.

   5. LRLenny Rachitsky5:37

      Let's start broad and set a little foundation. You have a really interesting way to describe what a marketplace business even is. So Ramesh, what is a marketplace business, and also why is data so important and such an integral part of building a successful marketplace business?

   6. RJRamesh Johari5:53

      You know, it's, it's interesting when people sit down and think about, say, Airbnb. What does Airbnb sell? The average person says like, "Oh, that's, that's pretty obvious. Airbnb sells rooms." I go there to book a room I want to stay at, right?

   7. LRLenny Rachitsky6:04

      Mm-hmm.

   8. RJRamesh Johari6:04

      Other people say, "Oh, what does Uber sell? Uber sells me rides." I'd use Uber when I need to get a ride from somewhere to somewhere else.And in some sense, you're not wrong. I mean, you go there, that's a platform to get these things. But that's not what the platform is selling.

   9. LRLenny Rachitsky6:16

      Mm-hmm.

   10. RJRamesh Johari6:16

       That's a really important distinction. There are people on the platform that are selling that to you. The hosts on Airbnb are selling you listings. The, you know, the drivers on Uber are selling you rides. But Uber and Airbnb are selling you some- the taking away of something, which is a weird thing to think about. What they're taking away is the friction of finding a place to stay. They're taking away the friction of finding a driver. In economics, we call those things transaction costs. You know, when you take Econ 1, you learn about markets and how supply meets demand and we get prices out of that. But what you don't learn until like Econ, you know, 201, is that markets don't always work. And one of the reasons markets don't always work is because we have what are called market failures due to the presence of these kinds of frictions. So like what's a market failure? It's that, you know, Lenny wants to get from Palo Alto to Burlingame and he can't do it. Why can't he do it? 'Cause he doesn't have anyone to drive him. Well, why doesn't he just call someone to drive him? Well, who's Lenny supposed to call? Who are those people? Are they out there? Are they willing to drive him right now, right at like, you know, 10:00 AM on a Friday? Are they willing to take him somewhere? When I wanna stay somewhere when I'm traveling, a friction is who's willing to give me their room? I mean, in principle, there's people who are willing to let me stay in their living room, but I don't know who they are. So those are frictions and what the marketplaces are selling you is taking the friction away. That's what you're paying them for. And it's an important observation because what that means is the marketplace's customers aren't just the people buying the rides or buying the listings. Actually, the hosts are Airbnb's customers and the drivers are also Air- uh, Uber's customers, right? So both sides of the marketplace are the customers of the platform. Both sides depend on the platform to help the platform take that friction away 'cause just like you want a place to stay or you want to ride, the driver is at Uber because he wants to earn money by taking people places and the host is on Airbnb 'cause they wanna earn money by selling their listing.

3. 8:10 – 11:21

   The role of data science in marketplaces

   1. RJRamesh Johari8:10

      I think this concept that we're making money by taking transaction costs away is such a fundamental idea that's misunderstood around marketplaces that when you're an entrepreneur starting a marketplace or thinking about your business model, I think you can be wildly off if you forget that that's the thing that's fundamentally your value proposition. You know, and then you asked about the role of data and, and more broadly data science in, in marketplaces. So it's an interesting thing, right? Like the example I always love to give are the ancient Agoras, you know, in, in Greece or, or like Trajan's Market in Rome. You know, when you look at pictures of these things, what really stands out to me is the rock. I mean, these things are made of stone. It's not like you were gonna move a booth from one place to another place without moving a lot of rock from one place to another place. So you flash forward to 2023 and here we are with technology undergirding pretty much every kind of commerce now. And it means we can architect and re-architect the marketplace kind of on the fly, and we really are doing it all the time. And these frictions that are getting taken away, they're getting taken away because of data and data science. So I really wanna like highlight kind of three pieces of this for people, which I kind of, I- I want you to think of them as a cycle, but to start with, let's just lay them out one at a time. Okay. One of them is finding people to match with. So that's the problem of I wanna stay somewhere, who is out there who's willing to let me stay with them on a given, you know, timeframe? And then if I'm a host, I have a listing, who is out there who's willing to stay at my place when I have it available? So that's finding matches. Then there's making the match. And so here, you know, going back to my time at oDesk, you know, a- a big problem that we dealt with there was if I've got multiple applicants to my job, who should I hire? Who should I interview, right? It's a common problem we face in the real world, but now it's all remote. I don't meet these people in person. All I've got is this application they submitted to me. I need help triaging that. Okay. So that's helping make a match out of possible partners you can match with. And then finally, we make matches. Well, what do the matches tell us, right? I mean, if you stay somewhere in Airbnb, you learn something about the host, you learn something about the listing, the host learns about you too. And you know, that's all information that the marketplace should feed back in. So this is where we get to rating systems and feedback systems, even passive data collection, right? Did you leave your booking before you were supposed to leave? Well, maybe that's a sign that something didn't quite work out the way you wanted to work out. So that's passive data collection. Did you leave five stars? That's active data collection. Get all this back in. And what does that do? Well, that lets us do a better job finding potential matches and make potential matches in the future. Every single thing I just said, finding potential matches, making matches, and then learning about those matches and then, you know, cycling back again, that is the data science of marketplaces. And I feel like every marketplace that you could think of, you know, in any vertical has those three problems to deal with and relies on algorithms and data science to help them solve it. And in turn, that is the, I think really the underpinning of taking those frictions

4. 11:21 – 16:43

   Common flaws of marketplaces

   1. RJRamesh Johari11:21

      away.

   2. LRLenny Rachitsky11:22

      Many founders try to start a marketplace business, think about marketplace opportunities where they don't exist, and there's often these like recurring failures of types of marketplaces that just don't work in a area. I was just writing a couple ideas down while you're chatting, like cleaners, getting cleaners as a marketplace doesn't seem to work ever. Car wash, there's a classic failure too. Like getting tasks done for you on demand as a marketplace seems to not often work. So this might be a too big of a question, but I'm just curious if anything comes up of when someone is starting a marketplace or thinking about starting a marketplace business, what do you find are the most common flaws in like, this is probably not going to work as a marketplace?

   3. RJRamesh Johari12:01

      That is such a fantastic question and I wanna preface, uh, what I say with a couple of comments. So one of them is that, you know, uh, I've worked with a lot of different marketplace companies, but anything I say is pertaining...... you know, to something more sensitive. I may not name the company, uh, just over the course of the podcast. Um, but the other more important thing I wanna say is that, you know, I'm a professor at Stanford and there's a reason I'm not, like, a successful scared- scaled entrepreneur of marketplaces, and that's because I probably haven't unlocked the key to exactly the question you asked. But nevertheless, I have some thoughts on it. The most important one is this. What I found talking to people who wanna start what they think is a marketplace is that they think too much about a marketplace before they're a marketplace. That, in my view, is the biggest failure mode. You know, you mentioned specific things. Cleaners, you know? I, I wonder about that, right? Is it about, something about the cleaning industry? It possibly is, right? I don't claim to be an expert on the, the microeconomics of the cleaning industry. But often, it's not that. It's that I thought I was building a marketplace from the beginning, and that's not the way the world works. So I'm gonna give you one vignette of this that I really like and that's Urban Sitter. So when Urban Sitter first... Urban Sitter is a babysitting marketplace, okay? We can, we can talk about kind of their whole life story, but I think what's most interesting is really the early days. And in the early days, what, what I found interesting, the way I found out about them actually, is that we were stuck looking for some help, and I found out about this new platform where kind of the, the, the clever thing was, you know, when you used to hire a babysitter, this is like pre-Venmo days, you needed cash on hand 'cause when the babysitter's done at the end of the day, they're usually like, you know, high school students or something, they, they wanna get paid. (laughs) They're not gonna take your IOU that you'll, you know, send them some check in the mail the next day. And unfortunately, you often don't have cash. They don't take credit card. They're, you know, they're high school students. That was an incredible friction to address, which is literally just, "We accept credit card payments for babysitting." That's it, right? Now, from there, what happened is they took advantage of Facebook networks between parents and babysitters to build trusted introductions. So like, let's say my sitter wasn't available. I get to know sitters in the Facebook network of, of that sitter, right? And once they overcame that first thing to get some liquidity onto their platform, they could move towards asking, "How do I solve for these frictions that I talked about earlier? How do I solve for helping people find potential matches? How do I solve for people making those matches," right? You can't do that when you don't have liquidity on your platform. It's silly to tell someone, "Hey, I'm really gonna help you find all those drivers out there, even though I only have three drivers on my platform." That's not a friction you're solving for. So in their example, as they evolved, they actually shifted their monetization model away from billing specifically for this friction of allowing you to pay with credit cards instead to now billing for how you were interviewing and contacting sitters. They had kind of a, a two-part plan for that. You know, one with like a pay-as-you-go menu, one with a more of like a subscription option. But the, but the key thing was either way what you were paying for now was finding potential babysitters, not paying them with a credit card. That wasn't the key thing anymore, right? So what's the moral there? The moral is a marketplace business never starts as a marketplace business because what we think of as a marketplace business is something which at scale is removing the friction of the two sides finding each other. But when you start, you don't have that scale. So when you start, you had better be thinking, "What's my value proposition in a world in which I don't have that scaled liquidity on both sides?" And you know, that's bespoke. It, it means different things. In, in the case of oDesk where I started, that initial thing was that remote work is a weird thing because basically you've somehow gotta know that this person who you're not next to is doing what you're asking them to do. And so the initial value proposition of oDesk was to provide tools for workers to verify they were working the hours and doing the things that they said they were doing, you know, screenshots and, and various kinds of tracking. And then in return for that, to be able to provide guarantees on both sides, right? So now the workers could say, "Hey, I worked what I said so I should get paid." And the employers could say, "Oh, you know, I actually see that you worked what you said and so, you know, I, I feel comfortable that I got what I paid for." That was the initial value proposition, is resolving a trust issue at a remote scale, right? At that point, liquidity isn't the game. It's asking, "What's a, what's a problem that people in this space are facing that I can deal, deal with when I'm not a scaled marketplace?" So I, you know, again, with the cleaning industry, I can comment on that (laughs) from personal experience, but otherwise, I think that's the way I would think about it. It's almost never about building a marketplace when you're building a marketplace.

5. 16:43 – 20:26

   Why every founder is a marketplace founder

   1. RJRamesh Johari16:43

   2. LRLenny Rachitsky16:43

      That's (clears throat) very similar to the advice I always give marketplace founders is, like, 90% of your problems are gonna be non-marketplace-specific problems. They're gonna be the same problems any startup is gonna have. Like, how do I grow? It's gonna be, like, the same things you need to do.

   3. RJRamesh Johari16:58

      So you know, one thing you said was that's what you tell marketplace founders. And something I've actually pressed hard on in, in my own way of thinking about this is that maybe we shouldn't talk about the concept of a marketplace founder. Really, there's founders. And I think every entrepreneur... Uh, I mean, one way to think about it, right, it's very hard to think about a human business endeavor that has not been disrupted by the potential for transactions to take place online. And if that's the case, it means literally any founder is a marketplace founder. It'll be a choice they make after they grow as to whether they wanna build a platform. I mean, to take a very, you know, a very hot recent example, no one in their right mind would have thought of OpenAI as a marketplace, right? But OpenAI is a marketplace now. They may not wanna call themselves a marketplace, but they have plug-ins. Plug-ins are flooding that, that platform. I don't know, you know, if, if people have played with it. I mean, it's not, it's not an easy thing to find the plug-in you need for what you want to do, and that really is a two-sided thing now. There's the plug-in creators and there's the users, and, and they may believe it, they may not believe it, but they are a marketplace. So I think a different way to think about it is every founder is a marketplace founder. It's gonna be a choice they wanna make for themselves of whether they wanna become that platform. That's, I think, one. And two is, because that's the case, I think one of the other challenges I find founders struggle with-... is you'd never wanna... You don't wanna over-commit your future. And what I mean by that is that you're building up trust, that you're building up a sense of what kind of business you are in your early days. If you believe that this kind of platform future awaits you or market, you know, market platform future awaits you, you're, there may be choices you're making early on that are tying your hands later. A, a great example of this is, you know, when oDesk started, it was because the tools they were providing were for ongoing monitoring of work. It's a very natural thing to say, "You know, we'll just take a constant fraction of the dollars that cross the platform." That all works well and good until, you know, after you become mature. Some of these relationships between worker and employer last a long time, and most of the value is generated now, not so much because they're able to track each other, 'cause the trust is now there, but because they found each other, because they're able to build that relationship through oDesk. And that meant that, you know, the longer that goes on, the less value the platform is adding into that relationship, right? And, but you're still pulling 10% of all the dollars. So what does that lead to? A word that, you know, most marketplace CEOs know well is disintermediation, which is where you were intermediating between the two parties and now they... Disintermediation means, you know, that essentially they're like, "Hey, we don't need you anymore." Right? Favorite example is we had some stuff delivered from Ikea by a Thumbtack worker once and, you know, my wife is like, "Oh, you know, thanks a lot. You're so reliable." He's like, "Hey, great. Here's my business card. Ever need me again, just call the number on the back," right? And that was it. Like Thumbtack got their one lead gen and then, you know, we, we didn't, we didn't need the platform anymore. And I think this issue for oDesk meant that what, after they merged with Elance and became Upwork, uh, they had to think a little bit about, okay, what's the monetization strategy we wanna use? How do we address this issue that longer term relationships may disintermediate? Does that mean you need a pricing plan that actually takes that into account, right? So early commitments, in this case to like a particular pricing scheme, particular monetization, can really tie your hands as you then realize later you actually are a platform.

6. 20:26 – 20:58

   How Substack increased value to creators by driving demand

   1. RJRamesh Johari20:26

   2. LRLenny Rachitsky20:27

      I really like this message. It makes me think about Substack actually, which started as a, just a platform for newsletter writers.

   3. RJRamesh Johari20:33

      Mm-hmm.

   4. LRLenny Rachitsky20:33

      And then they're like, "How do we make this more valuable?" 'Cause they take a cut of everyone's revenue. And they've actually invested heavily on helping drive demand to writers, for example, me. And at this point, over 80% of my subscribers come from Substack's network. And so they've built this marketplace element exactly as you're describing, where they just found, here's a pain point. Writers need more subscribers. How do we help them drive subscribers? So they figured out all these ways to create demand.

7. 20:58 – 22:24

   An example of overcommitting at eBay

   1. LRLenny Rachitsky20:58

   2. RJRamesh Johari20:58

      That's like a really positive story, right?

   3. LRLenny Rachitsky21:00

      (laughs)

   4. RJRamesh Johari21:00

      Where they, they, they managed to actually expand the frontier of their business-

   5. LRLenny Rachitsky21:07

      Yeah.

   6. RJRamesh Johari21:07

      ... by enabling that network. You know, for every one of those, there's unfortunately, you know, a lot of negative stories. I mean, one that I think is very painful is how eBay sort of had a lot of challenges with its seller community as it introduced, you know, more and more fine-grained kind of sources of fees. And I think a lot of that... Eh, I mean, there's many, many treatises at this point written on eBay and their history and, and how they got to the point that they're at. But I think one kind of simple thing I do want people to think about there is that the sellers on eBay who had, who had matured with the platform, who had grown with it, had come to develop certain expectations about what their lives on that platform would look like. And it's understandable because a lot of these businesses, they had built their livelihood on that platform. Tha- that, that was their entire business. So when you now reach in and you say, "I'm going to completely change the rules of the game in which your business model operates," you know, from the perspective of those sellers, that's, that's a breaking of a social contract that's been developed over a very long time. And s- you know, I, I love the Substack example because that's like, "Hey, let me amplify our social contract," right? But, but I think for every one of those, there's an eBay warning sign that you can also, uh, you can trap yourself a little bit.

8. 22:24 – 25:52

   An easy litmus test for marketplaces

   1. RJRamesh Johari22:24

   2. LRLenny Rachitsky22:24

      Just to close the loop on this really, I think, important point. A lot of people listening to this are probably, "I'm a marketplace founder, I'm building a marketplace," are gonna hear this and be like, "Oh shit. Maybe I need to rethink-"

   3. RJRamesh Johari22:35

      (laughs)

   4. LRLenny Rachitsky22:35

      "... how I think about what I'm doing." What would be your piece of advice to people like that? Is it focus on the friction point? And it may be a marketplace solution, it may be a managed marketplace, it may be you own the supply. Is that the advice? Or how would you... What would your advice be to someone that's like, "I'm building a marketplace"? What, how should they reframe their thinking?

   5. RJRamesh Johari22:54

      Let's go back to kind of thinking about this concept of a marketplace as reducing friction, right? So the litmus test I like to give to someone who claims to me that they're building a marketplace business or they're a marketplace founder is, you know, do you have what I would call scaled liquidity on both sides of your platform? What does scaled liquidity mean? What it means in lay terms, and, and that's... By the way, I, I am a data scientist and I love to think about these quantitatively. But, but fundamentally, like if it doesn't pass the smell test, then you don't have to keep going with the data science. And the smell test is scaled liquidity asks, "Do I have a lot of buyers and a lot of sellers on my platform? Or do I only have one of these two? Or do I have neither?" Okay? If you don't have both, it... You could call yourself whatever you wanna call yourself, but at this moment in time, you're not a marketplace. All right? If you have won, congratulations. You've won the game on one side of the market, and now you can, you can, if you want, you have a choice point. You can lean into growth on the side that you're doing well with, right? You got a ton of users, ton of buyers, great. Lean into it, get more buyers. That's one option. It's, there's no shame in not being a marketplace. There's, there's, it... You know, scaling a business is scaling a business. If that's the way to do it, do it. If you decide you want to be a marketplace, then at that moment when you've got a lot of buyers but not a lot of sellers, or a lot of sellers and not a lot of buyers, the choice you're facing is, how do I take advantage of having that one side scaled to attract the other side? We can talk more about that, but there's a lot of ways to kind of hack that, to think about how... So, you know, to take Uber as an example, right? They would walk into a new city.And, uh, one thing that, that, you know, Uber was, was kind of commonly known for doing, this is back in the days when really Uber Black was the only service, is they'd just hand out coupons for free rides at kind of, you know, events, parties, things like that, to take people home. And that was a way of saying, "Hey, we're subsidizing the drivers in this city. That's our scaled side. Now we're gonna use that subsidized driver base to attract riders." Okay? So that's like how do you get that flywheel going and, and again, you know, many people have written about how to take li- liquidity, scale liquidity on one side and use it to attract the other side. If you don't have either side, don't worry about it. Don't worry about being a marketplace. Worry about scaling one side. And in that world, it- it's opened up... It, it opens your visibility up completely into the advice of many, many, you know, startup advisors, right? People who have advice not so much about scaling a marketplace, but about scaling a startup. And I really... I wanna say, you gotta let the ego go on, at that point, right? Like, it's fine to articulate to people that your vision of the future is to be a platform or marketplace. As I said, virtually every business is gonna have that option at some point in, in you know, the modern tech-enabled economy anyway. So you're not saying something people don't already know when you tell, you know, an advisor or an investor that. But I do think you need to be humble enough at the starting point to recognize that it, there's no sense in talking about a marketplace if you don't have scaling on either side yet.

9. 25:52 – 28:02

   Thoughts on employees vs. contractors

   1. RJRamesh Johari25:52

   2. LRLenny Rachitsky25:52

      And then it becomes a question of a business model u- uni- unit economics of can I build, say, a DoorDash not as a marketplace? Can I just hire a bunch of people delivering, right? It's like a... Is this even possible in a different route?

   3. RJRamesh Johari26:06

      Yeah. That's a, that's a great point. You know, one of the things I think that's useful for people to think about here that you're raising, at some level it's kind of, uh, tied up, I think, with that question of whether I should have employees or kind of contract or freelance, uh, work on one side of the marketplace, and that's actually a pretty old question in economics. Uh, this distinct... The, the way we, we talk about it often is a distinction between a market or a firm, and kind of one of the interesting puzzles in, in economics, Ronald Coase is a famous economist who thought about this, is, well, you know, if markets are so efficient, why do we need firms? Right? 'Cause if markets are efficient in matching labor up with things that need to get done, why would you ever need a firm? And that's like one of the earliest recognitions that transaction costs are a real thing, and, and that's kind of one of the things that, that firms are solving for. And I love what you're saying, because what it's recognizing is, hey, for your frictions, the best resolution to that might not be to have a marketplace. It might actually be to have very tightly controlled labor. A good example of this actually, you know, Stitch Fix, I think one of the things that's cool about Stitch Fix is the experience that people had early on with stylists at, at Stitch Fix.

   4. LRLenny Rachitsky27:13

      I'm a happy customer, by the way. I think I-

   5. RJRamesh Johari27:15

      Yeah. (laughs)

   6. LRLenny Rachitsky27:16

      ... can fix stuff on.

   7. RJRamesh Johari27:17

      Yeah. I think, I think like one of the great things about that experience is it felt magical to have, have someone who kind of got to know you, right? But that depends on a, a relationship that doesn't feel like a freelance relationship every single time you're, you're going back. Another example that, that I would, I would pull out is pretty much any healthcare platform. So, you know, for example, for physical therapy, uh, you know, it'd be weird if every time you went to a physical therapy platform you just got randomly matched to whoever happened to be available then. So, so I think there, there's some, there's some curation that needs to happen of that relationship. Does that mean full employee? Maybe not. But it does mean you have to think a little bit about, exactly as you brought up, you know, what's the, what's the nature of curation of your, of your labor pool?

   8. LRLenny Rachitsky28:01

      Awesome. Okay.

10. 28:02 – 34:10

    How to leverage data scientists to improve your marketplace

    1. LRLenny Rachitsky28:02

       So let's come back to a point you made early on around the importance of data and the power of data in actually making your marketplace a lot more efficient and work more effectively. So say that you have a data scientist or a data analyst or someone that is helping you optimize your marketplace. Where do you often find the biggest leverage and opportunity for a data person to help you make your marketplace more effective?

    2. RJRamesh Johari28:26

       Th- this is an incredible question, right? Because I, I think I could answer it, you know, a number of different ways. Like, one question I think there that's kind of basic is just what should this person be doing?

    3. LRLenny Rachitsky28:37

       Mm-hmm.

    4. RJRamesh Johari28:37

       And I'm not... I'm gonna actually kind of evade that question a little bit. I'm gonna give some examples of what they could do, but I feel like that's one where context matters a lot. So as an example, you know, at, uh, ridesharing or grocery delivery marketplaces, pricing means actually what do you pay for that ride or what do you pay for that delivery, right? So that's, that's actually the price that's set at the moment you actually place the order. Just to be clear, by the way, if you order from DoorDash, I don't mean the price of the restaurant. I mean what do you pay to DoorDash, right? What's that, what's that fee? Is there a surcharge because it's, you know, surge or whatever, right? So... Okay. So that's a thing, right? But that's not really a thing in a marketplace where the platform's not setting the prices. So at Airbnb, really, hosts are the ones who are in charge of setting prices for their listings. So one answer to your question is if I'm in a place like, you know, Uber, Lyft, DoorDash, I wanna have good data scientists thinking about pricing, 'cause that seems like something which would be heavily dependent on the instantaneous state of supply and demand in my marketplace, right? So that's one type of answer is, well, do I need people, data scientists working on pricing? Do I need data scientists working on search? Why search? Because maybe in my marketplace finding the needle in the haystack is really the biggest, highest friction problem. So maybe I need a lot more data scientists working on search. That's what I'm gonna evade. Okay? I'm gonna focus more on something completely different, which is just a, a more philosophical point about what a data scientist does. So in a lot of companies today especially, a, a main thing that you ask data scientists to do is build what's called a machine learning model. Now machine learning model even already can mean a lot of things to a lot of different people. I'm gonna focus on something very concrete. You're asking them to predict something. When I started at, at oDesk, this is in, in 2012, one of the funny things about me is I started at oDesk because I had a, a academic career up to that point of about 10 years just-... you know, building mathematical models of things. I was not really very much of a data scientist up to that point. What I expected would happen is, I go to industry and I'd be told, "Hey," like, "Look how important data is." You know, and definitely my eyes were opened. And one of the first things I was asked to think about is, well, okay, someone comes to Odesk, posts a job. Workers apply to that job. Predict which of these workers is most likely to be hired on that job. That was the narrow question. Okay? And, and so why is that a good question? Because we have a whole awesome set of tools now to solve that kind of a problem exactly. How do we do it? We take a lot of past data of past jobs, past applicants, past hires that were made, and then we ask, you know, these crazy big black box algorithms, "All right, do the best job you can predicting who's gonna get hired on this job with these applicants." And we use that data to test how well these applicant, uh, these algorithms are doing. That's like machine learning in 30 seconds basically. All right? So, you know, we're working on this problem. Great. And then I kind of poked my head up a little bit. Like, oh, why are we working on... What's, what is this gonna do? Well, it turns out the reason these kinds of things are important is they get used to make decisions. Right? So what kind of decision do you make with that? Well, one thing you do is you say, "Oh, well, if I could predict who's most likely to be hired, then I should just rank people based on that." And that would be a good matching algorithm, right? That would be a good way to sort and triage people, uh, applicants for, you know, for employers when they're, like, screening, trying to figure out who to interview, who to hire. Great. Sounds pretty natural, right? And then, you know, you think about it a little bit, and this, this to me is really... It's, it's such a, such a passion project to get people to understand that this is why the humans in the loop that help us in, in businesses and making sense of data are so critical is, is the following problem. If you think about it a little bit, you realize, you know, what that algorithm is doing, it's really just picking up on patterns in past data, right? So yeah. That's great. This person's likely to be hired. But what we really want is something different. We're trying to add value by ranking people. So, you know, to give another example that's similar to this, when you're a marketing manager and you've got, you know, a crack data science team that's built a long-term value, lifetime value model for you, you're not gonna get in trouble with anyone if you send your highest value promotions to the highest LTV customers, right? Who's gonna, who's gonna blame you for that? 'Cause you're like, "Oh, yeah," you know, "this person's worth a lot and I sent them this promotion." You know, say that in your monthly report, nobody's gonna give you a hard time. But the problem with that way of thinking is, actually predicting what their lifetime value is isn't really the question. The question is, how much more are they gonna spend on my platform because I sent them that promotion? That's a very different thing. It's a differential rather than an absolute. I'm not interested in their absolute LTV. I'm abs- interested in the difference in their LTV because I sent them this promotion. And when you look at it that way, what you realize can happen is picking up on patterns because of good predictions, right? Finding the people that have high LTV 'cause you predicted that is very different than making good decisions, which is about saying, "The difference in their LTV is gonna be higher 'cause I sent them this promotion." I love this example because I taught a class here at Stanford, it was, like, an executive education class. We had, you know, all the executives from the company in the room and one of the people in the room was the chief marketing officer and I just asked this question. Like, "Hey, okay. Let's say you have this great LTV model. Who would you send the promotions to?" He's like, "Definitely the highest LTV people." And there's a CMO in the room and so, you know, it's like, it's a, it's a, it's a little bit of a delicate situation, like, pushing back a little bit, right? I do wanna be clear. There's reputational reasons you might do that anyway. I mean, I'm not trying to get away from that. But just to make the narrow point that

11. 34:10 – 35:27

    Correlation vs. causation

    1. RJRamesh Johari34:10

       predicting is about picking up patterns, but making decisions is about thinking about these differences. Now, why is that important? Because we learn in high school, correlation is not causation. That's a phrase everybody has heard all over the place. What does it have to do with this? Well, when we teach people to build machine learning models, we're asking them to make predictions, we're asking them to find correlations. Prediction is inherently about correlation. But when we ask people to make decisions, we're asking 'em to think about causation. If I make this decision, then will I actually increase the net value of my business, right? Will I have, by sending the promotion, increased the likelihood that this person is gonna spend more on my platform? And so the first and most important thing that I feel very strongly about and what would I get a data scientist to do is, no matter who they are, right? Even if it was that person in the weeds thinking about building this, you know, prediction model for, for hiring, get them to be thinking in the back of their mind always that their goal is to help the business make decisions and that the distinction between causation and correlation matters a lot. You know? We could talk a lot more about, how does that play out in, in terms of their day-to-day work? But at least at a starting point, you have to recognize that the first step is always recognition, that prediction isn't the same thing as making decisions.

12. 35:27 – 39:29

    Decisions that should be made using data

    1. RJRamesh Johari35:27

    2. LRLenny Rachitsky35:27

       So the takeaway here is help... As a data team and as a data scientist on the team has helped the business make predictions. Are there a couple more examples you could share of just, like, what is an example of a decision that you think they often should be making and using data to help them with?

    3. RJRamesh Johari35:41

       Maybe the right way, the right frame of reference for this and, and the word that an academic would use is cau- you know, causal inference, right? So what we're changing from is machine learning to causal inference. So let's think that through in a, a couple of different use cases that are related to that, that, you know, marketplace data science flywheel I talked about earlier. You know, finding matches, making matches, and then learning about matches. So finding matches, like you said, a core part of that is search and recommendation and, and each of those relies on rankings. So, you know, I wanna be able to rank order... Let's say I go do a search on Airbnb. I wanna rank order the different listings in the marketplace, right? At some level, it's true that what I'm trying to do there is I'm trying to just predict what are you going to like the most, right? But I think there's an important piece of that also, which is that I wanna think a little bit about the distinction between two different ranking algorithms-That's the real decision that's being made. And when I think about the distinction about two different, between two different ranking algorithms, I don't wanna be only comparing them in terms of how well they recreate the choices people made in the past.

    4. LRLenny Rachitsky36:46

       Mm-hmm.

    5. RJRamesh Johari36:46

       The way I'm really gonna evaluate those is in my market, does one of those lead to better matches or more matches than the other one? Right? So if Airbnb as a business, like what are the most obvious core metrics? It's bookings and revenue. So you're gonna wanna ask a very basic question. If I use the ranking algorithm Lenny just developed last night versus the ranking algorithm Ramesh developed last week, does Lenny's ranking algorithm lead to more bookings than Ramesh's ranking algorithm? And it's so important to put it that way starkly because that's so different a question than does Lenny's ranking algorithm do a better job of predicting over the last two years what bookings people made than Ramesh's ranking algorithm? Okay. So that's, that's the thing like, you know, at that level. Then, you know, we talked a little bit about, about ranking at the point of making a match, and I think that's where this kind of hiring issue popped up, right? Because in the end while we might have these predictive algorithms to rank who you're going to hire, that's not the important question. Interestingly, the important question is-

    6. LRLenny Rachitsky37:39

       Mm-hmm.

    7. RJRamesh Johari37:39

       ... actually to evaluate the quality of the match that's made. And we would do that through the next step of the, of that flywheel. We'd ask ourselves, you know, what ratings did they give back to that freelancer? Do they hire that freelancer again? So you're comparing two different algorithms not through their ability to recreate the past, but their ability to make matches in the future that can be objectively evaluated to say, "Hey, I increased the value of the business. I, I actually made better matches this way." You know, and then rating systems I think we could talk quite a bit about kind of a similar, a similar phenomenon there too.

    8. LRLenny Rachitsky38:12

       (instrumental music) This episode is brought to you by Eppo. Eppo is a next generation A/B testing and feature management platform built by alums of Airbnb and Snowflake for modern growth teams. Companies like Twitch, Miro, ClickUp, and DraftKings rely on Eppo to power their experiments. Experimentation is increasingly essential for driving growth and for understanding the performance of new features, and Eppo helps you increase experimentation velocity while unlocking rigorous deep analysis in a way that no other commercial tool does. When I was at Airbnb, one of the things that I loved most was our experimentation platform where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Eppo does all that and more with advanced statistical methods that can help you shave weeks off experiment time, an accessible UI for diving deeper into performance, and out-of-the-box reporting that helps you avoid annoying prolonged analytic cycles. Eppo also makes it easy for you to share experiment insights with your team, sparking new ideas for the A/B testing flywheel. Eppo powers experimentation across every use case, including product, growth, machine learning, monetization, and email marketing. Check out Eppo at geteppo.com/lenny and 10X your experiment velocity. That's geteppo.com/lenny.

13. 39:29 – 41:06

    Ramesh’s philosophy on experimentation

    1. LRLenny Rachitsky39:29

       Yeah, I would actually love to talk about rating systems, but there's kind of an implication in everything you're describing of running an experiment versus looking at what would have happened in the previous world. It's you've made a change, run an experiment, see if it actually makes an impact on bookings and revenue. And that leads me to a question I wanted to ask, which is, with experiments, there's kind of this classic challenge and always elephant in the room of if you just run a bunch of experiments, you're kind of gonna micro-optimize, lead to these local minima, local maxima, and you may miss big opportunities and big unlocks if you're just like extremely experiment-driven. You spend a lot of time thinking about experimentation. What have you learned or what advice do you have for people to either be less worried about micro-optimizing and missing something big or just finding a balance with running experiments but also creating opportunity to find a huge new opportunity?

    2. RJRamesh Johari40:21

       Yeah. First of all, I'm really glad you broached the E word. I was dancing around it, and, and I'm, I'm really glad that we talked about experiments, 'cause yeah, that, you know, one of the big lessons of this like recent conversation we've, we've been having is just how could you possibly know that difference without doing something like experimenting? Okay, so yeah. I am a big believer in experiments. I mean, I'm, I'll just lay those cards on the table. I, I, I love working with businesses that think experiments are important to helping make good decisions. Now all that said, I am also someone who feels pretty strongly about this exact issue that you're raising, which is you can't experiment your way out of everything. And one way, you know, that, one frame I like to give people is that although you might say you're an experiment-driven business,

14. 41:06 – 44:11

    How to find a balance between running experiments and finding new opportunities

    1. RJRamesh Johari41:06

       I, I, you know, some businesses will proclaim, "We literally test everything." But that kind of leaves aside a little bit is there's a lot of degrees of freedom in what it means to test everything, because ultimately what's getting built and tested are choices that are made through the organizational structure, the data scientists, the PMs, the engineers, everybody's on the, you know... Before we're running experiments, we're actually thinking about even what's worth experimenting, right? Like, what designs are we coming up with? So that's one, and the other big one is, how long do we run these experiments? Okay? That's a big choice. And what I generally believe, and I think there's a paper I'll, I'll, we can, we can link to later that I'll, that I'll point your readers to as well that... Not my paper. Uh, from some folks at Microsoft. What I generally believe is that we're risk-averse on both these two dimensions, that what people decide to test in a world that has promoted experimentation for everything tends to be more incremental by design. Okay? Because... And, and we'll come back to why, actually. I mean, answer the because in a second. So that's one, and two is people tend to run experiments for a long time, and probably longer than they should. Okay? Now what do I mean by these two things? So what's interesting to me about this dynamic is experiments don't live in a vacuum. Companies have incentives, and in, uh, in companies that really go all in on experimentation, one of the things that gets wrapped up in that is the incentives around experiments, because if you go all in on experiments, the common thing you'll see is data scientists get judged based on...... how many wins they had that quarter. And, right? How do you get more wins? Well, it's easier to get wins when you're being incremental. And because it's important to have wins, you have to run them long enough to demonstrate that they're really wins. Right? You're less willing to cut something off in exchange for trying something riskier. So the big lesson of this Microsoft paper, it's, it's called A/B testing with what's called fat tails, which in lay terms just means you're running a business where there's potentially big opportunities out there if you, if you look at kind of the effects of the experiments that you run. But there's a couple lessons there about both trying a lot more stuff that's, you know, not all risk-averse, and not necessarily running everything for so long, so really getting velocity up. So you can see that there's a big incentive problem there, right? Because the culture that says it's okay to fail big actually requires changing the terminology of wins. This is one of the things I hate most in A/B testing, I have to say. I get where it comes from. You know, experimentation was never historically in science about winners and losers. It'd be weird if, you know, Ronald Fisher, who's kind of the father of experimentation with, with his agriculture experiments, talked about winners. I, I don't think that's necessarily (laughs) how you talk about things. Experimentation is always very hypothesis-driven. It's about, what are you learning? And that's really an important distinction, because what it means is if I go with something big, risky, and it, quote-unquote, "fails," meaning that doesn't win, I... Nevertheless, if I was being rigorous about what hypotheses that's testing about my business, I'm potentially learning a lot, right?

15. 44:11 – 46:04

    Badging in marketplaces

    1. RJRamesh Johari44:11

       So a great example of this kind of thing is, you know, uh, that, that there's... A- an important feature of, of marketplaces is badging, right? So sometimes it's really important to have badges on your kind of top-rated profiles or whatever when, when people are searching. And without going too far into the details, a, a common kind of finding, it, it, you know, with badges is that, that badges you think are gonna be great actually turn out to be terrible. Mm-hmm. And one reason they're terrible is they focus too much attention on the badged folks and pull too much attention away from the unbadged folks, right? And if we judge that only in terms of winners and losers, you throw the baby out with the bathwater. You're like, "Oh, well, that badging idea was terrible." So, you know, ditch that, would... you know, no badges. But that's not what it's telling you. It's teaching you something about how inventory I- is being reallocated, how attention is being redirected through the badges. And you really wanna think not in terms of winning and losing but learning. So learning is a win, and I feel that that's a cultural thing, fundamentally. It's very hard to somehow attach dollars and cents at the top, to data scientists running experiments that, that fail but learn. And, and ultimately, I think getting into that space where you experiment more, meaning you don't run all your experiments for quite as long, and you accept the willingness to, to try experiments that are into the tails where you might fail bigger is a cultural thing. It's about saying that, that, you know, we're allowing that to be part of our social contract with our data scientists or, you know, actually our employee contract with our data scientists, that not everything is just about how many launches you had and, and, you know, how many wins there were. It's okay to say, "That's how I want to use experimentation." But if you're gonna use it that way, then I would say don't be a "we experiment everything" business. 'Cause then I think you need some other way to deal with these big, you know, changes that, that, you know, teach the whole company a lot but, but maybe can't fall into the, the incentives you've created for your data scientists.

16. 46:04 – 49:59

    The “superhost” badge at Airbnb

    1. RJRamesh Johari46:04

       This badging example is, uh, I don't know if you're referring to the Airbnb example, but I actually led the launch of Superhost at Airbnb, which is, like, the ultimate badge on Airbnb. Mm-hmm. And there was a lot of concern from the data team that it would destroy the marketplace, because they've built, as you described, this very well-crafted ranking algorithm with just, like, a prediction of, you know, you know, exactly as you described, which listings that guest is most likely to book and be successful booking. And then we're about to throw a badge on random listings in the results. Mm-hmm. And so this one data scientist on our team's like, "No, we can't do this. This is insane. We're gonna destroy it all." And we still went ahead with it. We ran an experiment showing the badge to some people, and some not, actually was no, no impact at all, which is... Like, Superhost itself had no impact at all on the business as far as I- we could tell initially, which is also bittersweet (laughs) because it felt like, with a, "Why did we even work on this thing?" There was, like, a slight benefit where hosts felt better. They felt more satisfied with being a host, but, uh, I went exactly through what you described. So, that's pretty funny. Without necessarily, like, going into the weeds on, like, the data science of Superhost, I think there's a lot wrapped up in what you said. I, I, I guess another thing I'll say is that I'm a big believer that you don't, you don't throw your understanding of the business out the window when you process experiment results. And it's partly, partly, I guess, what I mean by this is data science is really about accumulation of evidence. It's never about one finding in isolation. And so another kind of trap, I think, is to sometimes say, "Well, I hit stat sig on my A/B test. You know, green light, it's all go." Like... And, you know, I think, you know, you had Roni Kohavi on your show, and he made a similar point that, that there are different levels of evidence. And, and it's just having an outlier A/B test that goes against everything you believe about your business doesn't mean that you somehow- Mm-hmm. ... have controverted all your knowledge. And I think that's one side of it. The other thing is you can't always measure everything that's important, that's needed to really develop, like, a full sense. So with Superhost, right, one of the things that's hard to measure is the long-term impact of Superhost. Because in the short run, Superhost causes a rebalancing of inf- inventory. There's gonna be winners and losers. Part of Superhost is actually about retaining hosts that get the badge over a longer period of time. Recognizing that hypothesis actually says something about maybe how long the experiment needs to be run or what kinds of data analyses need to be done. And in the end if you can't do that, you can't run it long enough, or you can't do that data analysis due to sparsity of data or lack of data to, to address the question-It matters what you bring to the table, right? What are your beliefs about that? So, what I like to tell people to do there is, I like to push people to be what's called quantified rather than data-driven. Which is, okay, fine, some things we can't measure, right? But you know, maybe you've got a leadership team with different beliefs about what they think the retention value of Superhost is gonna be, and they might be all over the place. You can process your experiment results in the context of these competing beliefs. It's almost like a prediction market kind of a thing. And, and start asking, well, okay, like if this is what we believe about our business, this is what the data's telling us out of the experiment. Let's put those two together and ask, is this, is this enough for us to make the bet that we're still gonna go with it, even though maybe that short-term test you ran was flat?

    2. LRLenny Rachitsky49:31

       That's actually exactly how I think of Superhost, looking back. It was a great idea. I'm really happy. It, I can't even imagine Airbnb without that, even though there's no evidence, at least initially, that it made any impact. I'm guessing they looked at it again and maybe there's something that came out of it. But even if it had no impact, it just feels like it made the marketplace better, and that was a big learning for me. Just like, it doesn't need to always drive a metric that you can measure. There's just like, this is the way it should work.

17. 49:59 – 52:41

    How marketplaces are like a game of Whac-A-Mole

    1. RJRamesh Johari49:59

       So one of the reasons the thing you said happens is because marketplaces are a little bit like a game of Whac-A-Mole, okay? And what I mean by that is like, so narrowly in the context of Superhost, because you're redirecting attention to some hosts at the, at the expense of other, it's not even obvious if bookings can really go up. Okay? Maybe you get lucky and maybe you get a bunch more bookings. One reason you probably wouldn't expect that in the first place is there's only a limited number of Superhosts. How many more bookings are they gonna be absorbing 'cause of all this extra attention? And you're taking attention away from other people. Without doing any data analysis, my prior would've been that booking should probably go down, right? And like one example that I came across, uh, with one of the companies I worked with that, that I love is, uh, you know, we, we were working together over a period of time and, and in, in a month we looked at some of the data and it suggested that our new, you know, supply side was having a pretty bad experience.

    2. LRLenny Rachitsky50:52

       Mm-hmm.

    3. RJRamesh Johari50:53

       It's like, oh, we gotta do something about this. So what we decide to do is build some custom bespoke features that were really gonna direct them to more experienced folks on the other side of the market. Good. And then, yeah, lo and behold, you know, pretty soon those metrics start to look better. But then we're looking at it, we're like, wait a second. Now, you know, the existing folks on the other side are having a worse experience. So you kind of whiplash around. You're like, oh, wait a second, we better do something about that. So we take them, we try to match them up with the more experienced folks, and now suddenly month after that you're like, you know, wait a second. And, and your metrics just keep moving around. And that's because the Whac-A-Mole game here is ultimately a lot of marketplace management is moving attention and inventory around you. Sometimes you get lucky and you really expand the pie for everybody. But I think, uh, Sirbas Solan, who, who was CFO at- at Upwork, uh, that I got to know there, and then went to Thumbtack later, he had this line when he came to visit our class that I love, which is, "You know, you have to recognize when you run marketplaces that many of the changes that are most consequential create winners and losers, and rolling with those changes is about recognizing whether the winners you've created are more important to your business view than the losers you've created in the process." And it's a hard reality because nobody likes to articulate the idea that a feature change is hurting some of the people in your marketplace. But, but because of this, this fundamental constraint baked into how marketplaces work, many of the things that we would choose to do and the re- reallocation they create can't necessarily create observed pie expanding wins in the short run. Uh, you're often making bets that that's where you're headed, partly through the reallocation that you're doing right now, you know? And so I think that's what's interesting about Superhost to me, is it partly points to thinking about what's the objective you would've defined, the metric you would've defined in the short run that captures this idea of a trade-off.

18. 52:41 – 55:43

    How to shift an organization’s focus from impact to learning

    1. RJRamesh Johari52:41

    2. LRLenny Rachitsky52:42

       That's a great way to think about it. I wanted to come back to this idea you're sharing of maybe you should run experiments more quickly, not wait for stat sig, have a culture of learning versus impact. In practice, it's very difficult because people are measured by impact, there's performance reviews, there's promotions, there's how much impact did this team drive? We're gonna look at their experiment results. You've worked at a lot of marketplace companies, a lot of different companies. Is there anything you've seen about just like how to, something you could do to help the company shift and actually work this way while also recognizing success and who's doing great, who's not, which team's driving impact, who's not?

    3. RJRamesh Johari53:22

       Interestingly, it's actually sort of an active area of research for me now. What I mean by active area of research is I care a lot about the incentives-

    4. LRLenny Rachitsky53:29

       Mm-hmm.

    5. RJRamesh Johari53:29

       ... that we create for data science through how we set up reward mechanisms. So there's, there's a couple things I think that could be helpful that are maybe, you know, they're maybe a little bit less about, uh, like I, I, maybe I'm not gonna directly answer the question you asked, 'cause I think that's a hard one, right? I think I recognize that measurement on impact is critical. Well, let me, let me answer that actually from the most obvious way first. I think there's a cultural issue here that's really critical. You know, one of the things I often find is that my PhD students, our PhD students here often go off and get, you know, great data scientist jobs. And in one sense they're doing amazing stuff. They apply really technically sophisticated methods. But when I look at kind of the problems they're working on, they're often more at the margins of the business than they should be. And it's, it's a cultural thing. It's basically because if you're measured narrowly on impact and that's all anyone sees around you, then it's very hard to engage with the creative aspect of, of, of business change and- and the strategic aspects of business change. So the cultural aspect there is, I think it's partly incumbent on the leaders to expect something more of their data scientists. And what I mean by expect more is that you expect them to do more than deliver narrowly defined, statistically rigorous results to you in their reports. You're actually expecting them to talk also about what they're learning about the business in the process. So, so where that's headed is this, this concept of being hypothesis-driven, which is like the technical phrase. What does that mean? Again, in a more lay sense, what it means is...... tests aren't going to be defined only in terms of winners and losers, but each test should also say something about what will we learn about a business flow, a funnel? Preferences of the, of the, of the guests, preferences of the hosts, right? Uh, what will we learn about their demand elasticity if we're changing prices around? You know, these kinds of things. So it's possible to articulate in an experiment doc, a launch doc, you know, what is the, what are the hypotheses that are being tested? So that's one thing I would say, is like just culturally setting the norms that learning is part of the discourse and it- it's expected actually, I think is, is important. But the other thing I would say that's, that's, you know, maybe a little bit more about kind of programmatically like what could, what could a, what could a team, a data science platform team do?

19. 55:43 – 57:50

    Frequentist vs. Bayesian A/B testing

    1. RJRamesh Johari55:43

       You know, a funny thing about experiments is that we throw past learning away effectively, and this is just an artifact of how we analyze experiments, that the methods used, the statistical methods used typically, you know, p-values, confidence intervals, these fall into a branch of statistics known as frequentist statistics. And the idea behind frequentist statistics, you know, without being overly technical is just I let the data speak for itself. There's no beliefs brought to the table about where that data came from. But if you think about this in like a company in A/B testing a company, it's like a weird thing, right? Because I might have run 1,000 A/B tests in the past on this exact same, you know, button or call to action or color, and now I'm gonna completely ignore that and focus only on this. So there's a, there's ways to take the past into account to build what's called a prior belief before I run an experiment, and not take the data from the experiment, connect it with the prior to come up with a conclusion of like, okay, in light of the past plus this experiment, what's it telling me about the future? And that falls broadly under the category of what's called Bayesian A/B testing. So that's one of the things I think can help culturally, weirdly, it's like a super technical thing, but I think it can help culturally because what it's doing is it's now rewarding people for contributing information to that prior. And, and I think it's then becomes possible to say, "Oh, like your experiment that failed actually moved our prior." And that's an important thing because by doing so, you're now altering kind of how we're going to think about this flow or this pricing plan in all future experiments, right? So there's like an information positive externality, uh, positive network effect that's generated for the rest of your business if I can somehow encode what you learned into those, the analysis of future experiments. So this is one thing I, I, there's a, a strong connection between the culture and incentives of A/B testing and the ability to actually incorporate past learning into these prior beliefs.

    2. LRLenny Rachitsky57:35

       I love that you're doing research in this area. We should bring you back when you've completed it and have the, uh, the ultimate answer for everyone to change how they operate.

    3. RJRamesh Johari57:42

       Yeah. One of the great things about professors is we never complete anything and never have ultimate answers, so.

    4. LRLenny Rachitsky57:47

       Oh, boy.

    5. RJRamesh Johari57:47

       Yeah. I'll do my best though. (laughs)

20. 57:50 – 1:01:55

    The idea that learning is costly

    1. RJRamesh Johari57:50

    2. LRLenny Rachitsky57:50

       (laughs) This touches on a really interesting concept that you shared with me around how learning, just learning isn't free. People think that they could just learn a bunch of stuff and there's not a cost to it. I'd love for you to just chat a bit about what that means.

    3. RJRamesh Johari58:02

       Let me start with an anecdote that I just, I just absolutely love this anecdote. I, I use it every year in class. So yeah, I was talking to, to a real estate platform and they, they have a, they had a marketing, uh, uh, data science manager who's basically, you know, responsible as, as many marketing managers are for allocation of ad spend across different channels. And what they discovered had happened at the end of the year is, um, in one hand the team had done great, but the manager had held out some subset of, of arriving visitors, not shown them any of the innovations they were making-

    4. LRLenny Rachitsky58:39

       Like a holdout group?

    5. RJRamesh Johari58:40

       Yeah, what... Exactly. What's called a holdout group in experimentation. And the important thing about this holdout, it wasn't authorized, like that, that's not the way things are supposed to work. There's, they've got their ad spend, allocate out your ad spend, great. So at the end of the year, you know, they looked at the holdout and they're like, "Wow, that cost us like, you know, a couple million dollars." It's something in that range. And it's like not a trivial amount of money, like what's the deal? What were you thinking, basically. And, and of course the answer was, "Well, I get that I cost you that much, but number one, now you know what my team's worth. And number two, you would never have had that answer-

    6. LRLenny Rachitsky59:13

       Mmm.

    7. RJRamesh Johari59:14

       ... unless I'd done that on my own," right? Now, why is that so powerful? I think what's, what I find so interesting about experiments is that when you don't know something, it seems not even a question that you would allocate some of your samples to, to all options, right? Treatment and control. Like I have two different ways of doing something. I don't know which one's better, so of course I'll give some samples to each of them. After the fact, you're like, "Oh, treatment was better. What the heck were we thinking? Why did we give all those samples to control," right? That doesn't make any sense now. There's this great Seinfeld clip where, you know, they, they get the bill at the... He, he mentions getting a bill at the end of a, like, large luxurious meal and people stare at the bill. "We're not hungry now. Why did we order all this food?" Right? So it's the same thing. Like, I mean, you know treatment's better now. Why'd you waste all those samples on control? And, and I think that is such a powerful observation that you have to put yourself in the frame of reference of when you didn't have the answer. And at that moment, what you're essentially saying to yourself is that it's worth paying to learn the answer. I, I think it sounds obvious the way we're saying it now, or this, this anecdote of the marketing manager and the holdout sounds obvious, right? What's culturally not baked in, I think, uh, is that idea... The... And, and the reason I say it's not culturally baked in, by the way, is because of the language of winners and losers. 'Cause if we use that language, what we're implicitly saying is that we wasted time when we ran an A/B test on a loser. If I reward you for shipping winners, then what I'm really telling you is, all the time that you spent testing out failures was wasted time. And I think, you know, of course, yeah, like you, you don't wanna keep data scientists around who regularly are just generating failures. That's not my point. But my point is there's a disconnect there. On one hand, we can all look at the story of this marketing manager and chuckle at it, right-And yet, every day we're instantiating language and processes that are reinforcing that same theme. Which is essentially trying to say to you, if you're wasting samples on things that don't ultimately end up being a winner, then that is a, the act of doing so is a failure. So I, I really feel, you know, that that, that idea that you have to pay to learn is, is again, it's a cultural thing, but it's also an education issue for, you know, businesses are populated by people of all stripes, not everybody comes from a data science or experimentation background. And this idea that, that learning is costly is not natural actually. It's, it's not natural as, it's not as a matter of human nature, it's certainly not natural as a matter of running a business.

    8. LRLenny Rachitsky1:01:42

       I love that example of the real estate platform where it's like very viscerally clearly ex- like cost. There's like-

    9. RJRamesh Johari1:01:48

       Yeah.

    10. LRLenny Rachitsky1:01:48

        ... 100 they lost because they didn't-

    11. RJRamesh Johari1:01:50

        Yeah.

    12. LRLenny Rachitsky1:01:50

        ... roll out experiments to those groups for a long time.

    13. RJRamesh Johari1:01:53

        Yeah. Yeah.

    14. LRLenny Rachitsky1:01:53

        Such a good example of this idea in action.

21. 1:01:55 – 1:04:41

    The basics of rating systems

    1. LRLenny Rachitsky1:01:55

       You mentioned star ratings. I know you spend a lot of time on designing rating systems. Sorry, I didn't mean to imply star ratings. That's just one implementation. Rating systems in general. So maybe just to keep it focused, say a marketplace founder is trying to decide and design how they do ratings and reviews and things like that. What's a couple pieces of advice you'd give them for how to do this correctly? And is there a model, like a model marketplace you'd point them to, like, "Oh, these guys really do it really well"? And I know it's, like, super specific based on the marketplace, but is there one just like, "Oh, they really nailed it"?

    2. RJRamesh Johari1:02:30

       Oh, man. That's a tough one. I think I'll answer the second part first. I don't feel like anyone's really nailed this.

    3. LRLenny Rachitsky1:02:35

       Oh.

    4. RJRamesh Johari1:02:35

       Um, you know, I think there's a lot of innovation that's happened, but I think fundamentally, we're still playing with the same kind of tools that we had when, you know, eBay and Amazon, like, first started thinking about how to do rating systems ages ago. And part of the reason we haven't nailed it is because there's a lot of dynamics in play that lead to what's called rating inflation, where if you look at ratings over time in a marketplace, one of my colleagues, John Horton, who's a professor at MIT and, you know, has, has worked very closely with Upwork, we worked together when I was at, at oDesk, he was the staff economist there. He's written a couple of really nice papers with this empirical phenomenon that over time, you see the median rating inflating, let's say, on marketplaces like, like oDesk, like Uber, like any of these, right? And, you know, there's a lot of reasons for this, but one of them is just that there's a reciprocity issue, right? Which is, it's, it's effectively, you know, from your perspective, it's kind of costless if someone says to you, "Hey, like, please leave me a nice rating," right? And, and you know, if you're seeing them or you're interacting with them, you know, most people don't want to be mean. So that happens. No, p- but there's another aspect of it, which is norming. As the ratings in the marketplace go up, they get normed, right? So that now you have a condition, you're like, "Oh, a four star rating, I'm really screwing this person over." Whereas maybe when the marketplace started, you didn't think that. So definitely one thing that we worked on, uh, in our research was to think about re-norming the meaning of some of these labels. And re-norming could mean something like rather than, you know, the star ratings just being, you know, poor to excellent, the top rating is actually exceeded expectations, right? You could go one step further and you could say, "How did this compare to this experience you had in the past that you rated really highly?" And, and Airbnb had something like this in place, uh, where they, they would actually ask you to compare or, you know, or ask you questions about expectations, and I find that that's really valuable because it's easier for people to say, "That was good, but didn't exceed my expectations. I mean, that was good, but definitely not better than this amazing stay I had like two months ago," than it is to say, "Well, you know, I'm gonna ding this person and give them four stars." So that's one issue, and, and I think another thing I wanna point out, uh, for any marketplace founder

22. 1:04:41 – 1:07:14

    The problem with averaging

    1. RJRamesh Johari1:04:41

       is that something you wanna be really careful about is, is the concept of averaging, and, and what are the implications of averaging. And that, that's because a default for many marketplaces is to just average the ratings that people get. It feels very natural, right? Like, Lenny's got five ratings, let me average them. And that actually has some pretty important distributional consequences for the marketplace, distributional in the sense of who wins, who loses. And that's because if you're averaging and you're really established on a platform, think of a restaurant on Yelp with 10,000 reviews, it's irrelevant what the next review is. Doesn't matter. Nothing's moving at that point. If you're new and you break into that market and your first review is negative, you might be completely screwed. In fact, there was some early work on eBay that showed that your first, if your first rating's negative, that could actually im- immediately cause like an 8% hit on, you know, your immediate expected revenue, to say nothing of long-term consequences. Subsequent work has found that that's a, a significant indicator of potential exit from the platform-

    2. LRLenny Rachitsky1:05:39

       Mm-hmm.

    3. RJRamesh Johari1:05:39

       ... just because now it's very hard to find work. And, you know, some platforms do things like maybe they won't show your ratings until you've accumulated a few, but in the end, this kind of distributional fairness aspect of averaging is pretty significant. And one of the recent papers that we've written is trying to get platforms to think a little bit about that. There's ways to address that, interestingly, through the same concept of a prior. And the prior basically says, "Hey, if someone comes into the marketplace and instead of averaging them, I average them together with a prior belief, then maybe what that prior belief does is it says, 'Yeah, you got one negative rating, but maybe you got a little bit unlucky.'" And maybe my prior belief is something which actually pulls your rating up a little bit and allows me to, to still have you alongside others in the marketplace to give you a chance at, at getting work, you know, getting rides, et cetera. So I, I believe pretty strongly in this kind of, like, distributional fairness element of designing rating system, rating systems. I think it's been understudied and, and, you know, I, I'll say in general actually, I think rating systems are understudied, which to me is astonishing, because the biggest change from those Agoras and Trajan's Market elements of, you know, those kinds of markets, to me, the biggest change is that we get to see what happened with our matches. So if, you know, as a data scientist working on marketplaces, I feel like it's, it's incredible that more of us don't spend our time thinking about what we're learning from the matches and what these rating systems are telling us and what the impact of that is on who wins and who loses in these markets, on, you know, kind of thinking about, like, the social implications of these things. So that's something I'm, I'm pretty passionate about.

23. 1:07:14 – 1:08:55

    Double-blind reviews at Airbnb

    1. LRLenny Rachitsky1:07:14

       I, uh, also led the review system flows for a while at Airbnb and one of the things I'm most proud of is launching what we called double-blind ex- uh, reviews where you don't see the other person's review until you leave your review. And we...

    2. RJRamesh Johari1:07:28

       Yep.

    3. LRLenny Rachitsky1:07:29

       The intention was to create more honesty and more accurate reviews. It turned out the biggest impact was review rate went up because people get this email, "Ramesh left you a review. If you want to see it, you should leave a review." And that really increased review rate, which gave us more data, and it was a really fun experiment to work on.

    4. RJRamesh Johari1:07:44

       There's a great concept in the ratings, in the literature on rating systems called the sound of silence, which is this idea that, that there's a lot of information in, in ratings that are not left. Um, so Steve Tadelos who's a professor at Berkeley, he had a really nice paper, uh, with some folks at eBay talking about what they called effective percent positive, where rather than normalizing just by the ratings, they normalized by including ratings that, that weren't left. And what you found was this was much more predictive of kind of downstream, downstream kind of performance of a, of a seller. So there's a lot of information in that, in that like lack of a response, you know? So it's cool that you're able to get more of that out.

    5. LRLenny Rachitsky1:08:23

       So much easier just to not leave a review than leave a bad review, right? Like the downside to you is just much better. Oh, man, marketplaces are so fascinating. I could see why a founder would wanna be a marketplace founder 'cause it's just like such an interesting space and, uh, hearing your feedback of like, no, you're not a marketplace founder, let's think about the problem you're solving and it might be a marketplace, might change people's minds. Also, I feel like there's like a podcast episode in every topic we touched on. I know we just kind of scratched the surface a lot of things. I know you gotta run. Before we get to our lightning round,

24. 1:08:55 – 1:11:27

    How large language models are affecting data science

    1. LRLenny Rachitsky1:08:55

       is there anything else you wanted to highlight, touch on, leave people with that are maybe working on marketplaces, thinking about a marketplace?

    2. RJRamesh Johari1:09:01

       I think one of the high level points I would make and, you know, like you said, there's an entire podcast in this topic is that I think people want to imagine LMs and AI-driven data science automating out large parts of what it means to do data science in, in industry. And I think that's probably the wrong perspective. In, in some like mundane sense, that's true. It's easier for me to code than it used to be before, easier for me to develop visualizations than it used to be. I can make dashboards faster. So like programmatically, I think it's true in some basic sense, but, you know, what I believe pretty strongly and I teach data science here and I'm- I- my students are asked to use LMs and generative AI on a weekly basis on all their assignments. So I've got like an up close and personal read on this. What I believe very strongly actually is what AI has done for us is it's massively expanded the frontier of things we could think about our problem, hypotheses we could have, maybe things we could test. It's just an astronomical explosion of explanations and ideas and principle. And I really think actually what that does is puts more pressure on the human, not less. I think it's becomes more important for humans to be in the loop in interacting with these tools to drive the funneling down process of identifying what matters at, at all levels. That ranges from you're carrying out a data scientific analysis and now because you've got these tools, you can hypothesize 10 explanations, maybe 100 explanations. Which of those are you gonna focus attention on? What are you gonna tell other people to focus their attention on? To you're running experiments, you used to have 10 creatives you're testing for a marketing campaign, now you got 1,000 creatives you're testing for that marketing campaign. Maybe that completely changes the game of what it means to run an experiment, you know? What are you actually looking for now? Uh, how do you evaluate that you found something that was good enough? And, and I think these questions are not getting enough attention. I think people are looking for the automated tool that, that really cuts the human out, but what I've seen so far and again, you know, who knows by 2024, I might have a totally different answer for you. I don't think so, but at the moment what I see is that humans have actually become far more important to the productive data science loop, not far less.

    3. LRLenny Rachitsky1:11:16

       Such an important point. I feel like we need to add AI corner to this podcast where we always think about how does AI impact what we're talking about on this podcast.

    4. RJRamesh Johari1:11:23

       Yeah. I can see that. I can totally see that.

    5. LRLenny Rachitsky1:11:25

       Okay. We might start doing that.

25. 1:11:27 – 1:13:52

    Lightning round

    1. LRLenny Rachitsky1:11:27

       Ramesh with that, we've reached our very exciting lightning round. I've got six questions for you. Let's try to knock through 'em so you can go teach your class. Are you ready?

    2. RJRamesh Johari1:11:36

       I am ready.

    3. LRLenny Rachitsky1:11:36

       All right. What are two or three books you've recommended most to other people?

    4. RJRamesh Johari1:11:40

       When it comes to books, I have one I love that I start with always, which is How to Lie With Statistics. It's a tiny book, fr- by Darrell Huff from 1954, which is just for anyone that likes data at any level, it's like such a fun read. It's like it's- it's a great book. The second thing I recommend to people and actually this is true even for people who are not, uh, you know, not expert is David Friedman was a statistician at Berkeley who, who passed away in, in the 2000s, early 2000s and hi- his writing was fantastic in getting us to think hard about process, but wha- like why... You know, he, he was especially fond of what he called shoe leather statistics where you really got your... You know, you, you rolled your sleeves up, you got on the ground, boots on the ground, really getting in there, really trying to understand your data. His writing is fantastic, his explanations are fantastic. Uh, he has like a few different books at different levels I think people love reading. Most importantly what I like about it is he puts such emphasis on, on driving evidence and understanding of your processes that generate data. And, you know, I find often data scientists don't even look at examples, right? So like at oDesk, it meant are you looking at actual jobs and what the- what- what's actually going on in, in your product before you're trying to do data science on it. So I think that's like a Friedman, a Friedman insight, Friedman mantra and so his writing is great. The last one I was gonna mention has nothing to do with data science or anything. It's- it's called Four Thousand Weeks, uh, by Oliver Burkeman. I'm not like a huge like self-help type person, but I really like this book a lot. It's a little bit... I think it's a little bit stoic in its approach, like stoic philosophy, but it's... The b- basic point is you're only on Earth for around, um, somewhere in the neighborhood of 4,000 weeks.And, uh, you know, my wife and I have this term we call the infinite queue, which is like no matter what you think you get done on a given day, more stuff's gonna just keep coming in. And he basically says that recognizing that is liberating. 'Cause once you recognize it doesn't matter what you do, you're always gonna have too much to do, there's no point in stressing out about having too much to do. And just that, like, small shift of mindset then puts a lot more attention on, you know, the usual thing people worry about, which is, where do I want to prioritize my time? So he has a great way of writing about it, some concrete rules of thumb that help manage, you know, that- that way of thinking. And yeah, I- I think it's a great book.

Episode duration: 1:23:35