Master Economist on Strategic Quitting and Valuable Decisions on the Margin — John List

1. 0:00 – 0:15

   Intro

   1. TFTim Ferriss0:00

      [upbeat music]

2. 0:15 – 4:38

   Clawback incentives: paying upfront to leverage loss aversion

   1. TFTim Ferriss0:15

      John, welcome to the show. It's nice to see you, sir.

   2. JLJohn List0:18

      Thanks so much for having me, Tim.

   3. TFTim Ferriss0:21

      Now, I find your story endlessly fascinating, which means, of course, I have an embarrassment of riches in terms of materials and questions. We're never gonna get through all of it. But I [laughs] was struggling with finding a place to begin. There's so many options. Uh, but I thought I would just mention for folks, you have eight kids. Is that right?

   4. JLJohn List0:42

      That's correct.

   5. TFTim Ferriss0:43

      Eight kids.

   6. JLJohn List0:44

      Eight kids.

   7. TFTim Ferriss0:45

      Your grandpa, dad, and brother all truckers, if I'm not mistaken.

   8. JLJohn List0:51

      That's correct. All truck... Proud truckers.

   9. TFTim Ferriss0:53

      Proud truckers. I'm just adding a little bit of color here and there because, uh, certainly we're gonna get into many different nooks and crannies. I thought I would just start with asking you to describe what a clawback incentive is and how you have used clawback incentives with your kids.

   10. JLJohn List1:12

       [laughs]

   11. TFTim Ferriss1:13

       And incentives in general, what other incentives you have used, and then we'll, we'll use that as a jumping-off point.

   12. JLJohn List1:19

       The clawback incentive is an incentive scheme that actually reverses the way we think about traditional incentives. So in traditional incentives, I'm the chief economist at Lyft, the way I incent my workers there is I have them work really hard all year, and then at the end of the year, they receive a bonus if they do a good job. That's great. Now, what the clawback incentive is, is at the beginning of the year, I give them the incentive, but then I tell them if they don't perform, I will take away the incentive. And the idea behind the clawback is I wanna, first of all, show them the money, and then secondly, I want to invoke loss aversion. Loss aversion is something that psychologists have taught us goes as follows: If you own something, you really, really, really are hurt if you have to give it up. So the idea here is give somebody some money that will induce them to work harder, and when they do work harder, everyone's better off because they get to keep the money, and my firm is better off because they work harder. Now, where I've used that, I've used that in Chinese manufacturing plants. I started a pre-K school about 12 years ago. I used that with the teachers. So at the beginning of the year in September, beginning of the school year, they... some of them get the bonus, and then I tell them, "If your kids don't perform at the end of the school year, I will take some of it back." And lo and behold, what happens? They work harder and the kids learn more. Now, for my own kids, I've also done the same thing. Any good scientist will use his or her own kids as subjects.

   13. TFTim Ferriss3:23

       [laughs]

   14. JLJohn List3:23

       So I- I'm no different than that. A- as you mentioned, I have eight kids. Two of those kids happen to be twins, so I have the perfect treatment and control group.

   15. TFTim Ferriss3:32

       [laughs]

   16. JLJohn List3:33

       Now, some things like my oldest daughter had a very hard time going potty on the toilet. So what do I do? I give her an incentive, let's say a doll or something else, and I tell her, "If you go potty on the toilet, you get to keep it. If you don't, the doll is gonna go and sit in a different room, and you're not gonna be able to play with the doll." That's the clawback incentive.

   17. TFTim Ferriss4:01

       And what were the results?

   18. JLJohn List4:03

       The results are just remarkable. Whether it's school teachers, whether it's workers in a manufacturing plant, whether it's students, whether it's my own kids, just across the board, what happens is if you use the clawback, people work harder and everyone's better off in the end because they are more likely to get the reward, and you're happier because they've worked harder, so you will... you're happy to give them the incentive.

3. 4:38 – 8:00

   Incentives with kids: focus on inputs rather than outputs

   1. TFTim Ferriss4:38

      So hearing of your kids and the twins in particular, I have to imagine an unfair share of the experimental load lands on the twins.

   2. JLJohn List4:49

      [laughs]

   3. TFTim Ferriss4:49

      But, uh, you can disabuse me of that notion if, if I'm missing. But could you describe other ways that you have used incentives with your kids? Any, any stories or examples that come to mind?

   4. JLJohn List5:02

      So first of all, you're right that whenever I need a really good control, that will fall on the twins. But, but to be fair, I have enough experiments. I think about using experimentation to incent inputs. For example, when the kids are in third or fourth and fifth grade, I tend not to incentivize outputs because outputs are something that a person has a hard time controlling. But what they can control is the set of inputs. What I mean by inputs is how many books you read or how many hours you spend studying. Those are things that the child or the worker or whomever can control. Things like how well you do on the exam, a lot of exams, for example, are graded on a curve, so you can't control how well other students do in the classroom, so it's not really fair to give you, uh, an incentive based on that kind of output. So, so look... My incentives roll from in the classroom to on the baseball field to on the soccer field, but it's all the time it's using input incentives with my kids. Now, with my workers, whether it's at Lyft or I w- I used to be the department chair here at the University of Chicago's econ department, there I can't really observe inputs, so I have to reward outputs. And in the end, it's the organization is affected by the outputs, not the inputs.

   5. TFTim Ferriss6:41

      Could you give an example, and don't worry folks listening, I'm going to expand beyond the family system. But I, I, I like to start [chuckles] with something that many listeners will identify with [chuckles] and then we'll move in all sorts of directions. Are, are there any specific examples that you could give of incentivizing and changing behavior or molding behavior with your kids?

   6. JLJohn List7:05

      So let's think about studying. When I think about a clawback, at the beginning of the week, I will tell them, "If you work intensively for an hour each night, Monday night, Tuesday night, Wednesday night, Thursday night, Friday night, you can keep this incentive that I will give you on Monday morning." Now, that incentive can be a pack of baseball cards for my boys. It could be a pass to the movies for my daughters. But the key is give them the incentive early in the week, and then they will perform all week. For example, an hour of studying math every night is something that happens all the time in my household, that at the end of the week they end up keeping the incentive because nearly every time they meet the goal.

4. 8:00 – 13:15

   Do clawbacks backfire? Durability and side effects over time

   1. TFTim Ferriss8:00

      Hmm. I have a question about the, the clawback incentive and harnessing the loss aversion. And, and part of why I was looking forward to chatting with you is that I think it's minimizing it to say sort of al- along the lines of, say, Freakonomics. I know you've had a lot of involvement with Professor Levitt and have been featured regularly on the Freak- Freakonomics blog and elsewhere. There, there are many assumptions that we make or beliefs that we hold which turn out not to be terribly supported by data [chuckles] when you actually run experiments. And as an example, when I was doing prep for this conversation, I read about, and, and I might, I might get the specifics slightly off, but your examination of matching donations with-

   2. JLJohn List8:50

      Mm-hmm

   3. TFTim Ferriss8:50

      ... charitable organizations and asking them if matching works. The answer is, of course, yes. Do you have data to support it? No, not really. But looking at, say, one-for-one matching, you donate $1, we match a dollar, or one to two, one to three, and it turns out matching does work, but the ratio is less important than just the matching mechanism itself, right? It, it would seem. And we can certainly explore that more, and if I screw it up, please let me know.

   4. JLJohn List9:18

      Mm-hmm.

   5. TFTim Ferriss9:19

      On the harnessing of loss aversion, you know, as an employer myself or as someone who's hoping to have kids soon, I'm not gonna c- probably g- not gonna catch up with you unless I adopt a lot.

   6. JLJohn List9:29

      [laughs]

   7. TFTim Ferriss9:30

      But I imagine in my mind that workers will respond well when they are given a bonus at the end of their performance review period, whenever that is. But that over time, maybe in the long term and not in the short term, if people are given something that is taken away, that that might breed resentment or some type of learned helplessness where they'll just not attempt to put forth their best effort. Is that grounded in any of the data in any way or, or, uh, are there any side effects?

   8. JLJohn List10:05

      You know, that's a great question. When you think about using behavioral nudges or you think about using behavioral incentives, one very important issue that you always need to recognize and keep in mind is will it keep working over and over and over again? And then secondly, does it have any side effects? So in the case of the clawback, you could imagine that you run it one time and it works, and then when you try it a second and a third time, it depreciates. It really doesn't work that well. Or you could imagine that you hire a bunch of school teachers like I did down in Chicago Heights and give them the clawback, and within three months they all quit because they hate it.

   9. TFTim Ferriss11:00

      [laughs]

   10. JLJohn List11:00

       It's just too stressful and it's too burdensome on them. Now, those are two very important considerations for any incentive that we talk about. What we've found so far is in terms of the depreciation, it does depreciate a little bit like most behavioral nudges, but not a ton. So, so on that side, you don't have to keep creating a new mousetrap, that the, the real mousetrap and the first one, the clawback, works reasonably well, and it has over time for a long time. Now, the side effect one is also sort of interesting because what you find is that many workers actually like the clawback because they view it as a commitment device. And what I mean by that is, you know, most things in life you have to exert effort today, and you receive benefits in the future. So think about climate change. Think about your own health. You don't really wanna go to the doctor. You don't really wanna go to the dentist because it's a cost now, and the benefits are in the future. Think about why people drop out of school- You know, as a 16-year-old, there's a lot of cost now, and the benefits aren't for a long time. As humans, we do a really bad job at those types of issues, and that's why we have problems like climate change, and you don't take care of your health, and too many people drop out. What's nice about the clawback is it takes that problem and turns it around, because it takes the benefits from the end and brings them to the beginning. So you get the good stuff right away, and that serves as a commitment device for many people. So some people in our experiments will actually pay real money to have the clawback, to get to use the clawback. In the end, there are some of course that hate it. But there are enough that love it, that on that dimension, it actually turns out to be a doubly good incentive, because you not only are better as the owner, but the workers like it more too.

5. 13:15 – 15:33

   Inside a Chinese factory experiment: weekly bonuses and quality output metrics

   1. TFTim Ferriss13:15

      Question about the Chinese manufacturing, just speaking to the durability of the incentive.

   2. JLJohn List13:21

      Yeah.

   3. TFTim Ferriss13:22

      And I, again, I'm getting probably the specifics off, and you can correct me, but i- it seemed that it, it, it worked on the order of something like six months, not on the order of weeks, at least in the example that I read. And maybe, maybe that was just the sampling interval, so maybe it lasted for years for all I know. But my question is, what was the reward period? Does that make sense?

   4. JLJohn List13:46

      Makes perfect sense. So what we did in the Chinese manufacturing plant is we had weekly reward periods. So every Monday morning, they were given the bonus, and then that cleared on Friday afternoon.

   5. TFTim Ferriss14:03

      Ah.

   6. JLJohn List14:03

      So our experiment lasted six months, where we would move people in and out of clawback incentives. We'd put some people in the standard incentive. The standard incentive, of course, is work all week, and at the end of the week you get your bonus. And we were always comparing whether the clawback does better than the standard.

   7. TFTim Ferriss14:23

      And could you speak to inputs versus outputs and what your metrics were?

   8. JLJohn List14:29

      Yeah, absolutely. So in that example, all of our metrics were outputs. So the output in China was the number of non-defective items that came off the manufacturing belt, and each team had a goal, and that goal was if it was met, you got to keep your bonus. If the goal wasn't met, you had to give back your bonus.

   9. TFTim Ferriss14:56

      Hmm.

   10. JLJohn List14:57

       So in the manufacturing plant, as in many of our experiments that involve workers, we are looking at outputs. Now, when I talk about my kids, I want to incentivize an input, but it's really hard to do that in the workplace.

   11. TFTim Ferriss15:14

       Yeah, that makes sense. That makes a lot of sense. I mentioned donations earlier, and I'd love to segue to tipping, and I know you've [laughs] spent more time than most looking at tipping.

6. 15:33 – 23:01

   From fundraising field experiments to tipping: what generosity data can and can’t predict

   1. JLJohn List15:33

      Yeah, yeah. Do you want me to tell you the vignette about donations and then how I can pivot that into tipping?

   2. TFTim Ferriss15:42

      That's perfect. I would love that.

   3. JLJohn List15:44

      My research agenda in charitable giving actually started back in the late '90s, and a lot of people ask me, "Why did you get involved in working on the science of charitable giving?" And a lot of times they assume it's because of my altruism, and I tell them that's not the case. [laughs]

   4. TFTim Ferriss16:08

      [laughs]

   5. JLJohn List16:09

      What happened was I was an assistant professor at the University of Central Florida in 1998. An assistant professor means you're the lowest on the-

   6. TFTim Ferriss16:19

      [laughs]

   7. JLJohn List16:19

      ... on the pole, right, right? No- nobody gives a damn about you. You know, you're just trying to do the best you can. So I'm sitting in Orlando, Florida, in my office, minding my own business, and I hear a knock on the door. I open my office door, and there's the dean of the College of Business standing in my doorway. So I say, "Dean," uh, his name was Thomas Keon. "Uh, Dean Tom, how can I help you?" He says, "John, I want to start a center for environmental policy analysis, and I want you to help me start that center, and your first job is going to be to raise money."

   8. TFTim Ferriss17:07

      [laughs]

   9. JLJohn List17:08

      So I'm thinking, "Well, [laughs] what? I've never raised any money before. I'm a truck driver's son. What, what's going on here?" So he said, "Think about it and come back to me." So I thought about it for a few days, and I went to the dean's office and I told him, "This is a great opportunity and I will do it if you satisfy two conditions. One condition is I want to run the fundraiser as a field experiment because I wanna learn about why people give, what keeps them committed to the cause, and two, I need some upfront money." And that's what I kind of learned in those two days is I, I went out and talked to a bunch of fundraising specialists and they told me the key is to have upfront money. So the dean gave me $5,000. Whoop, big whoop, right? But importantly, he gave me a big list of potential donors Who I could ask to give money to the Center for Environmental Policy Analysis. Okay, now I'm in business.

   10. TFTim Ferriss18:24

       [laughs]

   11. JLJohn List18:25

       My, my research agenda has always been to use the world as my lab, and if you're ever trying to accomplish something, do it scientifically so you can not only figure out what works and why, but then you can get it right the next time. So I did this fundraiser, and what I first learned was I talked to dozens of fundraising gurus, practitioners, strategists, et cetera, et cetera, et cetera, and they all had rules of thumb. They all said things like, "Use a three-to-one match rather than a one-to-one match, because a three-to-one match will raise more money." And when I say three to one, I mean for every $100 somebody gives, the charity matches it with $300. That's a three-to-one match versus a one-to-one. You know, all these, all these great anecdotes. And each time I would ask them, "Can you give me the data behind these gut feelings?" "No, I can't." And I said, "Well, how do you know they're true?" And every person said, "Well, that's just how we always do it. That's how my boss did it." Now, I'm thinking, "Holy cow, this is 2 to 3% of the GDP of our nation is charitable giving," right? 2 or 3% of wallet is giving of money. Forget about the time. You know, add, add volunteerism to that, and you might get 7% of GDP. So it's a very important sector. But when I learned that it's mostly built on anecdotes, I said, "Man, we're in business."

   12. TFTim Ferriss20:18

       [laughs]

   13. JLJohn List20:18

       Because now I can use this as a, as a research agenda to explore generosity, and I can help charities use science to raise more money. So I've been at that for now close to 25 years, partnering with charities, helping them raise more money, helping them leverage science, and you end up learning some really interesting facts about people. For, like, for example, when you look at what drives a man to give versus a woman, [lips smack] uh, a man is driven a lot by the tax advantages that charitable giving gives you. That's called the price effect i-in giving. Whereas women are driven more by altruism and social pressure. So now these become two sort of important features that you might say, "Well, that affects charitable giving, and that's about generosity." But what about other parts of society like tipping? I-Is... Can, can we take what we learn about the generosity to, you know, the Sierra Club or the United Way or what have you, and apply those elements to what happens in the world of tipping somebody, whether it's an Uber driver or tipping a waiter or what have you? And it's sort of interesting because sometimes you can and sometimes you can't. Like, like one example is when you look at the charitable giving data. So I have the IRS data, so I have everyone's tax form from like 2008 to 2017. So I can explore all kinds of different cuts of those data, and I can tell you that as it turns out, at each income percentile, so if you say the average income earner in the United States is like $58,000, if you look at all people who make that amount of money, women give a lot more than men in that income bucket. So you look across all of the income buckets, and a fact is, is that women in every income bucket give more than men to charitable causes. So now you'd say, "Okay, that must be also true in tipping." It's not.

   14. TFTim Ferriss23:01

       Huh.

7. 23:01 – 28:07

   Uber tipping data: who tips, who doesn’t, and how anonymity changes behavior

   1. JLJohn List23:01

      In fact, it's just the opposite in tipping. So you can say, "Well, where's John getting his tipping data from?" I was a chief economist at Uber back in 2017, and TK, Travis Kalanick, came to me and said, "John, you need to help us keep the drivers in Uber because there's this crazy Delete Uber campaign, #DeleteUber campaign, that's killing us on the, both the driver and the rider side." And my solution back then was, "Let's roll out tipping on the app." So we rolled out tipping on the app. My team at Uber was called Ubernomics. So Team Ubernomics helped roll out tipping, and what we find there is across the board, men tip a lot more than women. So there's sort of this interesting dynamic that in charitable giving, where you do it face-to-face or you do it to be recognized, women are giving a lot more than men. But remember, on the Uber app, what happens is you do it anonymously and you don't do it face-to-face. And in that case, men actually give more than women in tips.

   2. TFTim Ferriss24:22

      Huh. What are some of the ... learnings around increasing frequency or amount of tipping from your, your many experiments?

   3. JLJohn List24:36

      There are sort of some interesting facts about what happens in that Uber car. Okay, so a first fact is that if your listeners want to join the 1%, a, a lot of times people say, "I wanna be in the 1%." Now, in this case, only 1% of riders tip every trip.

   4. TFTim Ferriss25:03

      Hmm.

   5. JLJohn List25:04

      That kind of blew my mind.

   6. TFTim Ferriss25:06

      That's wild. I never would've guessed that.

   7. JLJohn List25:08

      Yeah, because in the charitable giving game, like nine out of 10 people give to a charitable cause at least once every year. But when you think about the Uber experience, only 1% of people tip every time. And interestingly, roughly 60% never tip.

   8. TFTim Ferriss25:33

      Wow.

   9. JLJohn List25:34

      Now, in the middle of course, are the sometimes tippers. So we did this big data, j- j- just this, this big data grab from our experiments, so now we're talking about 23 million observations, a big data set. And we can look at a lot of averages, and what you find are things like men tip female drivers more than they tip male drivers. That's a fact. Especially when the female drivers are younger. So what happens is the older female drivers, say 60 years old and beyond, those female drivers are tipped the same as the male drivers in that age group. But the big difference is with the 21 to 25-year-old drivers, where females earn about 6% more in tips than the males do. And that it's all driven by male riders, which is sort of an interesting fact. [laughs]

   10. TFTim Ferriss26:36

       Yeah.

   11. JLJohn List26:36

       Now, now I'm gonna let your imagination, uh, Tim, go with that about why that happens. I don't wanna go into that here.

   12. TFTim Ferriss26:42

       [laughs]

   13. JLJohn List26:42

       I'll let your imagination go.

   14. TFTim Ferriss26:44

       Yeah. I don't have to imagine very long. [laughs]

   15. JLJohn List26:46

       [laughs]

   16. TFTim Ferriss26:46

       Yeah, cont- c- continue, please.

   17. JLJohn List26:50

       But there are all these kinds of facts like this. You know, the average trip, o- only about 15% of trips are tipped. And conditional on tipping, the average tip is about 10, 12% of the fare. So y- you do have things like a, a bigger bill is, is a bigger tip. That's what we have in restaurants too, where people tip 20%. But the big thing about tipping is people tip a lot more if it's face-to-face versus when it's anonymous. So the way we did tipping at Uber was you don't get asked to tip until after the ride is done and you're removed in space and time from the driver. A- and we did that on purpose because TK said, "John, I don't want tipping to be a tax on riders. I want it to be something that is truly done for exceptional service. We wanna separate it in space and time," and then that's what we got, is, you know, roughly 15% of trips are tipped.

8. 28:07 – 35:45

   Designing tipping prompts: presets, choice architecture, and avoiding “too high” defaults

   1. TFTim Ferriss28:07

      Now, if, uh, if I think to my restaurant experience, so I grew up working in restaurants, and the, the checks have changed over time, if we're talking about computer-generated bills/checks. And one of the changes, I don't know when this started, and I don't know if it's supported by data, I would hope so, it certainly affects, has affected how I tip, but it has been check boxes where it'll have, rather than making you do the math, it will say 15%, 20%, 25%, and there's a checkbox, and there's the amount, and then there's the total. And that raises the question for me of what types of user interface or flow experiments worked well, if you're able to speak to it. And I think you are, because I did, I remember reading about you getting a call from Jeff Bezos, and one of your conditions for possibly engaging with Amazon being the ability to publish your findings.

   2. JLJohn List29:12

      That's right.

   3. TFTim Ferriss29:13

      And, [laughs] uh-

   4. JLJohn List29:14

      [laughs]

   5. TFTim Ferriss29:14

      ... just to give a snapshot of that, you didn't take the job because there were gonna be trade secrets, they wouldn't be publishable. However, you helped introduce someone who was hired, and I think still works there, and he sends you a, what is it? A Christmas card every year-

   6. JLJohn List29:27

      [laughs]

   7. TFTim Ferriss29:28

      ... with his net worth at the bottom. [laughs]

   8. JLJohn List29:30

      [laughs]

   9. TFTim Ferriss29:31

      Something like that, which is outstanding. I hope he sends you some chocolates too.

   10. JLJohn List29:36

       He better send me a bag of cash the way-

   11. TFTim Ferriss29:38

       Yeah. [laughs]

   12. JLJohn List29:38

       ... Amazon stock has been doing.

   13. TFTim Ferriss29:40

       Yeah.

   14. JLJohn List29:40

       They gave me that offer, the stock was $7 a share, Tim. [laughs]

   15. TFTim Ferriss29:44

       Oh, ugh, ugh.

   16. JLJohn List29:45

       [laughs]

   17. TFTim Ferriss29:46

       So, so TK was open, as I understand it, to you publishing.

   18. JLJohn List29:51

       Yeah.

   19. TFTim Ferriss29:52

       So could you, could you share some of the maybe design experiments or changes or tweaks that turned out to be effective? It doesn't have to be Uber-specific, but it certainly could be.

   20. JLJohn List30:05

       Early on, Amazon, Jeff Bezos came to me and asked me to be their chief economist. And oh, boy, oh boy, oh boy-

   21. TFTim Ferriss30:14

       [laughs]

   22. JLJohn List30:14

       ... whenever I go back and look at that contract, it w- it was a contract that was heavily laden with options. And, [laughs] ah, Tim, Tim, Tim. So if I had to go back-

   23. TFTim Ferriss30:27

       [laughs]

   24. JLJohn List30:27

       ... I, I still think, I, I can honestly say this, I, I would do the same thing today. I, I, I did give up millions of dollars, no, no doubt about that. But i- in the end, I'm a scientist, and I've always thought that my calling was that to use the world as your lab is a great opportunity to make the world a better place. And I really felt I could help Amazon, I really did. But in the end, they just couldn't get over the hurdle of I needed to publish in academic journals as a scientist because I not only wanted to solve amazon.com's problems, I want to solve the world's problems. And, and I, I certainly get it, that their basic premise was, "Look, we want you to come in and help us make a ton of money." And at the time he was talking about something called the cloud, which I didn't even know what that was. In fact, he goes, "John, the, the two next big things are gonna be the cloud and the economics team, and you're gonna be the guy who hires all the economists." And then he says, "Let me tell you about this cloud thing." I was like... He goes, "You're gonna help me price the cloud." Like, I didn't even know what that is.

   25. TFTim Ferriss31:44

       [laughs]

   26. JLJohn List31:44

       And I, I just pretended I did. But [laughs] like, honestly didn't know anything about it. Okay, so TK calls, U- Uber calls, and Uber says, "We understand what happened at amazon.com because of Jeff Holden." Jeff Holden was at amazon.com back then, and then he ended up leaving for Groupon. He was at Groupon for a while, and then he went to Uber. And when TK wanted a chief economist, he ended up talking to his exec team, Jeff Holden's one of them, and Jeff said, "You know, there's this guy who we really liked at Amazon, and he wouldn't come because of this restriction or constraint that he said he's an academic and he wants to publish some of the work." And TK said, "Well, if he's good, let's bring him in and he can publish some of his work." And to his credit, TK stayed true to that promise. So not only around tipping, we, we have two academic papers about the economics of tipping. Questions like, how should you do the presets? You know, at the bottom of your app you say you have zero, $1 or $2, or should it be one, two, and four? Or should it be 0%, 5%, 20%? We have all those configurations. And kind of the bottom line is you want that. You, you want there to be a choice. You don't wanna have people fill in the number. You, you want there to be a box that you can check, but make sure the boxes aren't too high, because if they're too high, you're gonna get a zero.

   27. TFTim Ferriss33:29

       Too high in a dollar amount.

   28. JLJohn List33:32

       Too high of, like, $50, 100, 150.

   29. TFTim Ferriss33:35

       Yeah.

   30. JLJohn List33:36

       Or 25%, 45%, 70%. A lot of times people will say, "Those are out of my range, so I'm gonna do zero." So you don't wanna do too high. You don't wanna do too low, because then you might be leaving money on the table for the driver. So it becomes an art, and it's an art that you can only figure out with field experiments. And a lot of those, you know, 5%, 10%, 20%, those are individual specific. So that doesn't fit with just everyone. Sometimes the low, it means high for somebody else, or a really high set of presets is low for somebody else. So just like the price and, uh, and the various other pieces of the algorithm, an optimal tipping scheme will take account of the consumer's traits and how they've tipped in the past. Okay. So another line of research that we worked on at Uber was what I call the economics of apologies. And as you know, it was pretty rocky at, uh, at Uber when I was there. In fact, I think there's a, there's a new show coming out, right, called, like, Super Pumped or something that [laughs]
9. 35:45 – 43:49

   The economics of apologies: measuring damage from bad trips and what repairs it

   1. JLJohn List35:45

      So, so let me tell you this story quickly about apologies. Is that okay, Tim? Do you wanna hear that?

   2. TFTim Ferriss35:51

      Yes, please.

   3. JLJohn List35:52

      This is kind of a perfect example of, I, I think the leadership of Uber and, and how they were open to science. So I'm scheduled to give a keynote talk at our big annual meetings. They're called the American Economic Association meetings. So these are annual meetings that take place every January. Okay, great. So I'm on the schedule to give a keynote talk alongside a guy who ends up winning the Nobel Prize in development economics named Angus Deaton. He's there to criticize field experiments, and I'm there to be the, the supporter of field experiments. So I get picked up at my house here in Hyde Park, Chicago, 'cause the meetings are downtown Chicago. Just happened to get lucky that year. The meetings rotate to a different big city every year. Okay. So the Uber car picks me up, and as usual, I'm still working on my slides. But, but I, I look at my watch and it's noon, roughly noon, and I'm supposed to be on stage at 12:30. So the driver starts taking me down there, and we get about three-quarters of the way down, I look up, look at my watch, and I'm like, "This is perfect." I, I, I have to just finish this last slide. I'm gonna walk in. These slides are gonna be great. Next time I look up, I'm in the front of my house back in Hyde Park.

   4. TFTim Ferriss37:28

      [laughs] Oh, no. Oh, no.

   5. JLJohn List37:32

      And I'm like, "No. What the blank is going on here?" And the driver said, "Well, the app blinked, and Professor, you were so busy working, I didn't want to disturb you. I thought that you changed the destination 'cause you forgot something at home." I said, "No, take me back down to [laughs] to the conference." So okay, so I get there, I run in, everything turns out fine. But that evening when I got home, I ended up walking out to our garage that's, uh, separated from my home, and my wife always knows when I go out to the garage it means somebody's gonna have hell to be paid.

   6. TFTim Ferriss38:16

      [laughs]

   7. JLJohn List38:17

      So I, I call TK and I say, "Hey, TK, let me tell you about what I think of your blankety-blank app. And guess what, TK? I will never take an Uber again. I am just gonna simply take a Lyft." I said, "TK, you can call your chief economist a Lyft loyalist."

   8. TFTim Ferriss38:41

      Oh, wow. He must love that. Yeah.

   9. JLJohn List38:43

      [laughs] So, so I said, "You know what, TK? The worst part about the story is that I never received an apology." And he says, "Well, John, we haven't gotten to that yet." And I said, "We have now." So what my team did is we explored in big, big data sets how much business does Uber lose from bad trips. N- now, here you don't wanna do an experiment, because I don't wanna just give people bad trips on purpose. What I have to do is I need to find, for example, Tim, and then I need to find Tim's statistical twin in the data, and I need to find when Tim gets a bad trip, but at the exact same time and over the same route when his statistical twin gets a good trip. And then I can look at in the future, say for the next 90 days, does Tim who got a bad trip spend less money than Tim's statistical twin? What happens is you do. In fact, millions and millions and millions of lost revenue happened because of bad trips. Okay. So now I showed that bad trips cost Uber hundreds of millions of dollars. Now, what are you gonna do about it? That's where the science comes in. So we ended up developing a scheme where we did various apologies within an hour of when a person had a bad trip. It's like one apology could be a simply a basic apology in an email. "Tim, we're sorry about your bad trip. We know we should be doing better. Please accept our apology." N- that's one treatment. But another treatment is the exact same apology with a $5 coupon that, "Please take this $5 coupon for your next trip." What we find is we can undo a lot of the bad stuff of the bad trips, but nearly every time you need that cash coupon.

   10. TFTim Ferriss41:05

       [laughs]

   11. JLJohn List41:06

       [laughs] Words just don't matter, but a little bit of cash with the, with the apology ends up undoing about a third of the bad stuff. So now we're talking tens of millions of dollars. And true to his word, TK coulda said, "Look, that's a great trade secret. We don't want anyone else, including Lyft, to know about that." But TK let us write that up as a scientific paper, and now it's in an academic journal and we can all learn from it about the economics of apologies.

   12. TFTim Ferriss41:37

       I have to give TK credit, because you think about the, and you obviously, but as the fierce competitor he is, it's all the more impressive to me that he would allow that. And you think of the potential ripple effects of that not just within ridesharing, but outside of ridesharing. I mean, it, it, it hopefully translates to a better customer experience in different sectors, right?

   13. JLJohn List42:07

       Absolutely.

   14. TFTim Ferriss42:08

       That's part of the potential consequence of being able to publish.

   15. JLJohn List42:12

       You're right, and I think that TK really got, and, and Logan Green does too, and John Zimmer now at, at Lyft, you know, we have great papers coming out of there too. But they, they totally get that we're not only here to change the face of transportation, but we also have another gem called big data, and a lot of information that we can use and we can leverage to help some of societal's problems. Be attenuated. And, and that's the great thing, I think, about firms and corporations and organizations today when they're willing to do that, even though it's not in their financial best interest to do so.

   16. TFTim Ferriss42:52

       It's an incredible opportunity. I, I remember way back in the day, they ended up, I think, cutting this short, [laughs] but when, uh, you know, there, there existed the God view and the ability to look at all of this, all of this data, and you c- you could really learn a lot about human nature. And they would publish on the, on the Uber blog and, and elsewhere all sorts of correlations that were endlessly fascinating. But it's, it's really... When you have a big enough data set and you're dealing with individuals, and you're able to identify not only their behaviors but how their behaviors change with different incentives and UIs and so on, I mean, you're, you're really just looking at this gigantic Petri dish called humanity and human nature, right, at the end of the day.

   17. JLJohn List43:41

       Absolutely. Absolutely. You're, you're spot on.

   18. TFTim Ferriss43:44

       And let me ask you, if I can find this quickly enough, looking through my many notes here-

   19. JLJohn List43:48

       Mm

10. 43:49 – 50:30

    Nice in the lab, ruthless in the market: reputation effects and real-world behavior

    1. TFTim Ferriss43:49

       ... what you have learned about, uh, if, if that's the right way to phrase it, human nature. [laughs] And I wanna bring up car dealerships. So I, [laughs] I believe, and please correct me if I'm getting this wrong, but, "The behavioralist meets the market: Measuring social preferences and reputation effects in actual transactions." This was in the Journal of Political Economy, 2006. [laughs] But the quote that I, that I really liked, and this tells you I'm kind of a Hobbesian at, at heart, I guess, quote, "Very few people will not screw each other," List says. "There are very few nice people out there." Could you just [laughs]

    2. JLJohn List44:27

       [laughs]

    3. TFTim Ferriss44:28

       ... elaborate on that and f- kind of flesh out how that came about?

    4. JLJohn List44:33

       [laughs] So the paper you bring up is actually near and dear to my heart because, you know, one reason why I got involved in economics to start was, you know, back in the day in the late '80s, I was a baseball card dealer. So what that meant was I went to these large conventions to buy, sell, and trade things like Hank Aaron, Barry Bonds, or Ken Griffey Jr., whatever, baseball cards. And what I, what I noticed was everything that was happening in that convention was not always in accord with economic theory. And I thought that what I was learning in the classroom could actually be tested at these baseball card conventions. So I started to do it in the early '90s. That's, that's one of the very early work in economics that uses the world as the lab, and it's exploring things like, why do people trade? Why don't people trade? Why do people negotiate the way they do? Why do dealers give women higher price offers than men? Et cetera, et cetera, et cetera. You know, discrimination, gender pay gap, et cetera, et cetera. So along comes the early 2000s, and what I notice in economics is that there is an explosion of research on something called social preferences. And what that means is we wanna measure how much people care for other people, okay? So clearly they do. Clearly giving to charitable causes is two and a half percent of GDP in America, so clearly people are giving to others. But the types of estimates that these studies were showing is that, you know, you'll give up half of your wealth to some anonymous stranger. And I started to wonder, how far can you push these lab results if you go to a market where people are buying, selling, and trading for their livelihood? So what I did was I set up an experiment where everyone in the experiment knew that they were taking part in an experiment. Guess what happened? They all were really nice.

    5. TFTim Ferriss47:10

       [laughs]

    6. JLJohn List47:11

       And, and they all-

    7. TFTim Ferriss47:13

       Yeah

    8. JLJohn List47:13

       ... were exactly like the literature in economics said they should be. They're all homo be nice to everyone rather than homo economicus, right? So then what I did is I observed those same people in the marketplace when they didn't know they were being observed doing the exact same thing that I had them do in the experiment. Guess what they do now? Totally screw the other guy over.

    9. TFTim Ferriss47:46

       [laughs]

    10. JLJohn List47:47

        And they're about maximizing their own incomes. So what happened to all this generosity and all, all of this social preference, I'm being nice to other people? The market and the competition and the anonymity, that I'm doing this, and if nobody can detect me, I'm gonna actually do it. So then I added a twist to that paper. What I did is I had them act in the marketplace, and they did not know that they're part of an experiment, but now what they sell can be graded by a professional certifier. In there, you find that they start to be nice again. So what's happening is that they care about their reputations, and whenever you can check them on their reputation, they're nice. Why? Because they care about the long run, and they want repeat business. But when you can't check them on their, their reputation, or you can't check the quality of the goods they sell- They outright screw people over

    11. TFTim Ferriss48:52

        You know, this is, uh, makes me think about one of my sensitive spots with respect to Uber and ridesharing in general, which was how, and I'm not sure of the right kind of bias label to apply to this, but you would see in the news, you know, Uber driver this, Uber driver that, all this assorted bad behavior.

    12. JLJohn List49:18

        Sure

    13. TFTim Ferriss49:19

        Which statistically was a very, very small percentage, but it, the implication was that somehow, in the early days especially, that ridesharing was more dangerous or lower quality than taxi rides. But the fact of the matter was there was no data capture-

    14. JLJohn List49:36

        Yeah

    15. TFTim Ferriss49:37

        ... with taxi rides. There was no accountability, no rating, no nothing. And the, the costs and the events, the adverse events were largely invisible, right? So it's-

    16. JLJohn List49:50

        That's right

    17. TFTim Ferriss49:51

        ... just a very unfair comparison.

    18. JLJohn List49:54

        Mm-hmm.

    19. TFTim Ferriss49:55

        Well, let's, let's hop from car dealerships going to a place that is, is much more m- meta, because I, I admire your ability and anyone's ability to test assumptions, even if they lead you to some, some very unexpected, surprising, or even uncomfortable conclusions. And I have, as you can see, and people who are on video-

    20. JLJohn List50:21

        [laughs]

    21. TFTim Ferriss50:21

        ... have just, you know, [laughs] piles and piles of, of paper in front of me.

    22. JLJohn List50:26

        You are a scholar.

    23. TFTim Ferriss50:27

        Well, [laughs] I pretend to- I play one on the internet.

    24. JLJohn List50:30

        [laughs]

11. 50:30 – 55:16

    Critical thinking and theory of mind: anticipating others using “primitives”

    1. TFTim Ferriss50:30

       And, uh, let me, let me read a line from you, and I would love for you to explain what it means. Here we go. Just a few sentences. "A first step to rise up my critical thinking hierarchy is to practice theory of mind. People have a really hard time doing this, but once you do, you avoid all kinds of silly mistakes at home, in the workplace, and on the streets." So there are kinda two terms that I would love for you to define, and then you can expand-

    2. JLJohn List51:02

       Mm-hmm

    3. TFTim Ferriss51:02

       ... it in any way that makes sense. Critical thinking hierarchy, and then theory of mind, practicing theory of mind.

    4. JLJohn List51:10

       Critical thinking is a bit like creativity, right? You know it when you see it. It-

    5. TFTim Ferriss51:16

       [laughs]

    6. JLJohn List51:16

       ... it's very hard to define what critical thinking means. Now, in my work, I, I've written a recent academic paper on critical thinking. I view it sort of in two camps. One camp is how you use data and how you gather data to make logical decisions. A lot of times people make fundamental errors when generating data. For example, if I really think Uber is a great idea early on, every data point that I get back that says it's a bad idea, I put on the sidelines.

    7. TFTim Ferriss51:58

       Right.

    8. JLJohn List51:58

       And every data point that comes to me that it's a great idea, I say, "That data is right." That's called confirmation bias, and it's one of the most important impediments to figuring out if your idea is good or not. Okay. So we have biases in how we generate, accumulate, and interpret data. That's one kind of critical thinking. The other kind of critical thinking is more abstract, and this is where theory of mind comes in. So theory of mind means how well do you put yourself in the shoes of another person. So there's a, a theory in economics called game theory, and game theory was started by really old mathematician who's now unfortunately dead, named John Nash.

    9. TFTim Ferriss52:53

       John Nash, yeah.

    10. JLJohn List52:54

        John Nash. That's, that's the name of my dog too.

    11. TFTim Ferriss52:57

        The bar scene in, uh, Beautiful Mind.

    12. JLJohn List52:59

        Exactly. One, one of my favorite movies, Beautiful Mind, even though the bar scene got it wrong.

    13. TFTim Ferriss53:04

        Oh, really?

    14. JLJohn List53:05

        Yeah, they messed up the Nash equilibrium. The reason why is because at least one of them should have gone for the blonde.

    15. TFTim Ferriss53:12

        Ah.

    16. JLJohn List53:12

        So an equilibrium is, given everyone else's choices, would I change my choice? And if you wouldn't change your choice, that's an equilibrium. If you would change your choice, then that's not an equilibrium. Remember what happened in the bar scene. They all went for the brunettes even though they all wanted the blonde. So that was like a definition of not an equilibrium. But anyway.

    17. TFTim Ferriss53:37

        [laughs]

    18. JLJohn List53:37

        We'll, [laughs] we'll cut Ron Howard a break-

    19. TFTim Ferriss53:40

        Yeah, yeah, yeah

    20. JLJohn List53:40

        ... 'cause that's a great movie.

    21. TFTim Ferriss53:41

        Don't, don't, don't let facts get in the way of a good story now.

    22. JLJohn List53:44

        Exactly. [laughs] How could you write that bar scene at the Palmer House? Okay. So now theory of mind ends up being super important for the abstract part of critical thinking because your next move needs to be in part determined by how you think that person will react. And if you can put yourself in the shoes, whether it's do I, do I lower my price on Uber, and how would Lyft respond? Or should I spend more money in ads? If I'm United Airlines, how will Southwest respond? Et cetera, et cetera, et cetera. All of these kinds of examples. Now, theory of mind is how well do you put yourself in the shoes of another person and figure out then what your optimal action should be. N- now you see, Tim, when you start having kids, you're gonna have to use theory of mind to say, "Okay, how am I gonna get my kid to pay attention on the soccer field?" Or, "How am I gonna get my kid to pay attention in second grade math if they're really not interested at all because they're doing math at a seventh grade level?" And that's you need to put yourself in the shoes of whether it's your student, your worker, your, uh, collaborator, your adversary, whomever, and you need to then backward induct and act in an optimal way. That- that's theory of mind.

12. 55:16 – 1:02:46

    Poker as a training ground: randomization and an “exogenous” bluffing rule

    1. TFTim Ferriss55:16

       I have, uh, not surprisingly many questions about this. So the next question that leaps to mind for me is like, let's, assuming that theory of mind is, uh, or understanding and practicing theory of mind is a valuable skill with tangible benefits, how do you develop it, right? And I, I suppose if we wanted to speak broadly, a, a lot of folks would say, "Well, it sounds like you're practicing empathy." But then on, on the, on perhaps the other side of the spectrum, on the more concrete side of things, I would say, "Well, if I just gave you 1,000 bucks and made you play online poker and set a max amount that you could lose per game to ensure a certain number of repetitions, you would probably get better at predicting how others, or at least get better at attempting to predict how others will respond giving certain gambits and strategies and so on."

    2. JLJohn List56:15

       Yeah.

    3. TFTim Ferriss56:16

       How would you suggest people develop this skill, or are there different approaches that you might suggest people take, or books or resources, anything, for people who wanna get better at this?

    4. JLJohn List56:29

       So what I would tell the listener is to start with what an economist calls the primitives of decision-making. And what I mean by that is try to figure out what are the preferences of that other person, what are the beliefs of the other person, and what are the constraints of the other person, because all choice will be governed by these three primitives. So for example, if I know that somebody adheres to the social norm of never, ever tipping, then if I'm trying to do something to give them-- get them to tip, if I put myself in their shoes, regardless of what I do, they're never going to tip, so I should not spend a lot of money trying to convince them or induce them to tip because what is governing their tipping behavior is this strong social norm that they will never tip. So that's called a constraint i- in, in a person's world. So I think the first step is to try to figure out what are the primitives of the person's decision-making, and after you figure that out, you then say, "Well, what's my optimal action given that the person operates in this way?" You know.

    5. TFTim Ferriss57:49

       Mm-hmm.

    6. JLJohn List57:49

       A lot of times it'll be how much money do they have, do they have the mental capacity to actually solve the problem that I'm giving them. When I play poker, you know, I, I think poker's a hard game because it's really hard to randomize. So if you're playing against really good players, you need to first of all decide what, what is the fraction of times that you're gonna randomize, and then you need to do it-- You know, when, when I say randomize, I mean what fraction of hands are you gonna bluff? And then if you say it's, let's say, 10% of the time, you then need to figure out which 10% of the time. Now, humans are terrible at that. Humans are terrible at doing something in a random way, so what I do when I play poker, I use the seconds hand in a clock to say, "Well, if I'm gonna bluff, say, 10% of the time," then I look at the second hand and say, "If it's at a certain point in the clock, I'll bluff, and if it's not, I won't."

    7. TFTim Ferriss58:59

       Hmm.

    8. JLJohn List58:59

       And then that helps me because it's an exogenous machine that is not susceptible to my own biases. 'Cause when you ask somebody to, for example, create a random sequence of coin flips, you, you can ask a really smart person to say, "Create 10 coin flips in a row," and what they'll do is they'll say, "Heads, heads, tails, tails, heads." What-- They'll have nearly exactly five heads and five tails over the 10, and they won't have enough runs. 'Cause sometimes you have tail, tail, tail, tail. But when a human mind creates it, they don't do enough runs, and they do too close to 50/50. That's just how we're built. So you need an exogenous force to help you with things like that in poker or anything else.

    9. TFTim Ferriss59:53

       Do you play poker often? Have you played for a long time?

    10. JLJohn List59:56

        During the rage I played, and I ended up getting-

    11. TFTim Ferriss59:59

        Wait, did you say the rage?

    12. JLJohn List1:00:01

        Yeah, you know, the rage to me-

    13. TFTim Ferriss1:00:02

        Was that your rage or just a-

    14. JLJohn List1:00:04

        ... was sort of like 15 years. The explosion.

    15. TFTim Ferriss1:00:06

        Yeah, the World Series of Poker on television and all that.

    16. JLJohn List1:00:08

        Yeah, World Series was starting. Phil Hellmuth was on, Ivey, these guys. And I ended up flying to Australia to give a keynote in, um, a place called Alice Springs. It, it's way out in the, in the boondocks of Australia. I don't know if you've ever been there, but-

    17. TFTim Ferriss1:00:26

        I've never been to Alice Springs. I've been to, uh, I've been to Australia. Don't recognize Alice Springs.

    18. JLJohn List1:00:31

        Yeah, so I'm out there and, um, there's a casino there. And it so happens that the conference is at the casino. So as the keynote speaker on, I think it was a Friday night, they were gonna take me out to dinner, wine and dine me. That's typically what you do with a keynote speaker. So around 1:00 that afternoon I said, "Well, the dinner doesn't start till 8:00. I'm gonna join this poker tournament." And, you know, thinking I'm gonna be out, I, I won't make the next table, et cetera. It turns out that I get on this magnificent run. And I make the final table, and by this time it's 8:00, so the conference organizers-

    19. TFTim Ferriss1:01:14

        [laughs]

    20. JLJohn List1:01:14

        ... are standing around [laughs] saying, "Come on, John, we have to go." I end up making the final two. It was an Aussie woman and me, and I ended up... It was kind of an out-of-body experience, but I won the bracelet.

    21. TFTim Ferriss1:01:30

        Wow.

    22. JLJohn List1:01:31

        And, and it wasn't that I was very good, but I did a little theory of mind. I, I, I'm sure I got lucky a few times o- on the flop. Maybe on the river got lucky once or twice. But I ended up kind of retiring there. What-- I'm a lit- little bit like Michael Jordan, "Let's retire on top." So I got my Australian bracelet, and I haven't really played that much since. I just don't have time to play.

    23. TFTim Ferriss1:01:56

        [laughs]

    24. JLJohn List1:01:57

        But it's an interesting game that I think people undervalue skill.

    25. TFTim Ferriss1:02:02

        Mm.

    26. JLJohn List1:02:03

        Uh, because I, I think poker is a lot more skill. Obviously, there's a ton of luck, I get that, but there's a lot more skill than people realize. It's, it's, it's really a great g- especially if you have a long time series-

    27. TFTim Ferriss1:02:14

        Yeah

    28. JLJohn List1:02:14

        ... of tournaments. The, the, the best end up winning, and I, and I certainly wasn't the best. So I, I, uh, I kind of retired after that bracelet.

    29. TFTim Ferriss1:02:23

        I am fascinated by poker, and I've actually interviewed Phil on the, on the podcast. He's hilarious.

    30. JLJohn List1:02:29

        I bet he's hilarious.
13. 1:02:46 – 1:12:11

    Why ‘The Voltage Effect’ book: scaling failures, the ‘voltage effect,’ and replication advantages

    1. TFTim Ferriss1:02:46

       ... made into 21. Let's hop to scaling. So the new book-

    2. JLJohn List1:02:52

       Yeah

    3. TFTim Ferriss1:02:52

       ... is The Voltage Effect: How to Make Good Ideas Great and Great Ideas Scale. Why write this book? You have many options with how to spend your time. You have many different projects. I don't know how on earth you, uh, honestly have your hands in so many things. We, we haven't covered-

    4. JLJohn List1:03:14

       [laughs]

    5. TFTim Ferriss1:03:15

       ... 5% of it. Why, why write this book?

    6. JLJohn List1:03:18

       It's a good question about, you know, we should always, always, always evaluate how we want to spend our time because opportunity cost of time is, is very expensive. I was growing tired... O- on the one hand, when I go and give lectures, a lot of the audience, especially the younger audience, will ask me a question along the lines of, "You know, we've been fighting poverty for decades. We've been doing study after study after study to fight poverty. How come we're not making a dent in it?" And you can substitute poverty out for discrimination, public education, what have you, and we haven't really made a deep impact, even though scientists have been working on this for a long time. Now, that coupled with an experience that I had in Chicago Heights. So in Chicago Heights, I started a preschool from scratch with my friends Roland Fryer and Steve Levitt. Steve Levitt's Freakonomics author. Roland Fryer's a famous economist at Harvard. We, we start a preschool for three, four, and five-year-olds, and it took a lot of shoe leather, and we end up running this for four years, and I'm on the ground day to day to day running it, and I get great results. And then I go to scale it, and I talk to policymakers, and they tell me, "John, that's a great little study, but it's not gonna scale." And I ask them why, and they say, "It doesn't have the silver bullet." And I say, "What do you mean silver bullet? What, what, what does that exactly mean? I, I've been doing social science research for 25 years, and I've never heard this argument-

    7. TFTim Ferriss1:05:25

       [laughs]

    8. JLJohn List1:05:25

       ... that my idea won't scale because it doesn't have the silver bullet." And then they say, "Well, we just don't know about this, but every time an expert tells us they have a great idea, it always fails to scale." Okay. So then the, the third part of this trifecta was my wife and I are sitting down the street from the Uber headquarters on Market Street. We're sitting in a cafe, and I'm telling my wife about the Chicago Heights project and how it's so frustrating. I'm telling her about all these people who always ask me, "Why aren't we making a dent in poverty or discrimination?" And then I'm telling her about scaling at Uber, and she says, "You know what? For the next part in your career, you should take on scaling." And I said, "What the heck? Let, let's do it."

    9. TFTim Ferriss1:06:23

       [laughs]

    10. JLJohn List1:06:24

        I said, "Let's do it." And, and because all of these threads in my life, the White House, the creating the early childhood program, working at Uber and now Lyft, a- all of these had features of scaling. And then when you come to the realization that you only make big change at scale, you start to say, "Why don't we have science around scaling? Why is it the case that people write a book that say something like move fast and break things or throw spaghetti at the wall and if it sticks, good, or fake it till you make it?" You know, you, you say, "Where's the science behind that?" And there isn't any So my chore back then when I first started saying, "I'm gonna take scaling seriously," is basically I wanna do the science of using science. And I've, I've created science in the past, but we haven't created it in a way that is meant to scale. What we do is we say, "I wanna do a great idea in the Petri dish, and then after I'm done with it, I move on to the next idea." I don't take seriously from the beginning that if I wanna change the world, somebody's gonna have to scale this, and I need to understand the features that are important when scaling in the Petri dish itself. So all of that came together and I said, "Look, we're gonna change the world by understanding the science of using science."

    11. TFTim Ferriss1:08:08

        I wanna first just give voice to an observation, and that is thinking back to your story of the, the onstage debate, the, the pro-field studies camp represented by you, and then the, uh, I suppose the criticisms of field studies on the opposite side. It strikes me that one of the enormous benefits of field studies, and again, I don't know what I'm talking about, so please correct me if I'm wrong, is that you can in some respects get around the replicability crisis that is found in basically every nook and cranny of science because you don't have to write grants or raise funding for a tiny pilot study that will never be replicated. If you're working with an Uber, you have incentives aligned, you have, uh, incredible kinda longitudinal data, and you have millions of data points. I don't know. I, I just wanted to sort of lob that out there.

    12. JLJohn List1:09:09

        No, Tim, you're right. When you think about the credibility crisis or the replication crisis in science, it, it really goes along the lines of you had a camp of people do lab experiments. And a lab experiment is you bring a group of sophomores into a classroom, and you have them do a task that they typically don't do, right? A- and that's the, the experimental approach in psychology, and it was the experimental approach before, before I came into economics, it was largely do things in the lab. And what was difficult there is that if you wanted to generalize those results to the extra-lab world, a lot of the features of the extra-lab world that might matter in decision-making are set at different points of the dial in the lab. So, like, the first obvious thing is the world is not filled by, by rich white kids who are in the lab doing the experiments at the fancy universities, right? Right? The world is filled with truck drivers and all kinds of other people. So that's where it starts, but it doesn't end there because remember, anonymity or whether somebody's watching you or whether you have experience in doing something or whether people opt in or out of a market, right? Uh, people in a market tend to be the winners. The ones who you don't observe may have tried and they're gone. They're long gone.

    13. TFTim Ferriss1:10:51

        Yeah.

    14. JLJohn List1:10:52

        So there's selection into markets, too. So in the field, you have not only the right people, but you also have the right set of circumstances, and what you're getting to is right, that when you can partner with a government... I, I work a lot with governments, I work a lot with firms, but when you can get big data quickly at scale, you can, in a very fast way, find out, you know, did that result in Seattle on green cars. You know, a bunch of people in Seattle love green cars if you're, if you're a Lyft consumer, but they really don't like it that much in Akron, right? So it's, you can quickly find these things out and then you scale where you should scale, and you try something new where you shouldn't be scaling. And I think that organizations, whether they're for-profit or non-profit or governmental, these are unique opportunities that we're just scratching the surface. And one of the external benefits, what economists call an ex- a positive externality, is that it's gonna be a lot easier to replicate. And the firm has a really strong incentive to replicate because they don't wanna waste a bunch of money.

14. 1:12:11 – 1:20:49

    Scaling as an ‘Anna Karenina’ weakest-link problem: five vital signs and real examples

    1. TFTim Ferriss1:12:11

       Yeah, absolutely. So let's, let's come back to this mention of the silver bullet.

    2. JLJohn List1:12:18

       Yeah.

    3. TFTim Ferriss1:12:19

       So I've, I've read that you have said, "Scaling is not a silver bullet problem, rather it is an Anna Karenina problem." [laughs] Could you explain what you mean by that?

    4. JLJohn List1:12:29

       Yeah.

    5. TFTim Ferriss1:12:29

       And then at some point, we don't have to do it right now, but I would love for you to kind of map out a success case and then a failure case or a semi-failure case and sort of, uh, do an autopsy on, on both, if that makes any sense. But let's, let's begin with not a silver bullet problem. Scaling is rather an Anna Karenina problem. What on earth does that mean?

    6. JLJohn List1:12:53

       The typical policy maker argues two things. One, that the result in the Petri dish won't scale, and what they really mean by that is that there will be a voltage effect. So what I mean by voltage effect is you have a great result in the Petri dish, but when you scale it up, you turn that mountain into a molehill. Okay? So great result in the original research, scale it up, you get nothing. That's the voltage effect. Okay. Now, policymakers will say that truly scalable ideas need to have the silver bullet. Now, they're exactly wrong there. What I've found in my research and what I talk about in the book is that the silver bullet is really like if you have this one great characteristic, you can scale. It's like a, a best shot technology. It's-- scaling is actually a weakest link problem.

    7. TFTim Ferriss1:14:03

       Ah.

    8. JLJohn List1:14:03

       And what I mean by weakest link is... So go back to Tolstoy. Tolstoy starts Anna Karenina by w- which, w- one of the best first lines ever in a novel. "Happy families are all alike. Each unhappy family is unhappy in its own way." Scaling is identical to that. The way I think about scaling is scalable ideas are all alike. Each unscalable idea is unscalable in its own way. So now you can say, "Well, what do you mean?" And what I've found is that there are five major reasons why ideas don't scale, and that's what I call the five vital signs in the book. So your idea is scalable if it doesn't have one of these flaws, you see? So it's a weakest link. When I say weakest link, think about airport security. The airplane is only as secure as the weakest link. Think about your automobile. The automobile will work, but it breaks if you have a flat tire or if a piston isn't firing. That's a weakest link problem, because it only works as long as each of the features is at a certain level. That's how I want you to think about the Anna K problem. Okay. So now once you determine does your idea have these five vital signs, let's go through an example to help bring these vital signs to life.

    9. TFTim Ferriss1:15:43

       Yes, please.

    10. JLJohn List1:15:44

        One of my favorite scientists, Jonas Salk. So Salk is responsible for the polio vaccination. What Salk does is he first does, like a lot of great scientists do, he tries it out on his own kids.

    11. TFTim Ferriss1:16:01

        [laughs]

    12. JLJohn List1:16:02

        [laughs] He finds it works.

    13. TFTim Ferriss1:16:06

        Skin in the game. [laughs]

    14. JLJohn List1:16:07

        Phew. You got a lot of skin in the game. And then he finds that he can replicate that result, so those original results were not a false positive. That's vital sign number one. Does your idea have voltage? So Salk found that it does. It's not a false positive, his initial result. Now what Jonas Salk does is he tries it out on a lot of different kids. What he finds is it works for every kid. So now he knows that the audience, what I call know your audience or know your slice of the pie, he finds, wow, it works for all kids. That's vital sign number two. Understand how big of a piece of the pie you have or know your audience, and be, be true to yourself about measuring how big can this market be. Okay. Now, here is where Salk enters sort of, uh, thin ice, and that's understand the situation in which your idea can work. So think about COVID. COVID vaccinations have worked, but it's really hard to get the shot in people's arms. So like a lot of medications, medication adherence matters, and getting people to take the vaccination matters a lot. The medicine's good according to science. Now, how do you get it in people's arms? That's where this COVID vaccination scheme that we're up to right now has failed. It, it hasn't been able to deliver. This is where Salk is brilliant, because what Salk does is he leverages the healthcare system to deliver the vaccination. Okay, now how does he do that? Here's what happens. Tim, you don't have kids yet, but when you have kids, you're gonna be in the delivery room, kid's gonna come out, everyone's gonna be happy. You're gonna make sure your wife is healthy, the baby's healthy, and then the doctor's gonna give you a playbook, and the playbook is gonna say, "We're gonna give these vaccinations today. You're gonna bring your baby back in six months. It's gonna get some-- a checkup and some new vaccinations. You're gonna bring your baby back in 12 months, same thing. 18 months, same thing." The polio vaccination occurs during that time period. Any parent is always gonna do well by their kid and make sure that they bring their kid to the doctor checkups. So it's not an extra cost, and you just naturally get it. So now you've overcome a really hard problem about the situation here. Now, in Chicago Heights, the situation that I have is I hired 30 really good teachers. If I wanna scale that thing up, I need to hire 30,000 good teachers. And if I have to do that all in the same input market, say I have to do that all around Chicago Heights, I'm screwed, because I can't replicate the quality of the original 30 good teachers. That's what I mean by properties of the situation. The, the fourth Vital sign is understand spillovers or understand unintended consequences. In the Jonas Salk case, he's again got that in spades because the polio vaccination has these great externalities. Once you get it, you can't give polio to somebody else. The fifth vital sign is understand the supply side, and what that means is how much is it going to cost you to provide it? W- with medications, the, the first feature of a medication is most of our expenditures are up front, and then after you expend a bunch of money on R&D, you have what's called economies of scale. To actually make the vaccination or make the pill, it's really, really cheap. But you also need to get it into people's arms, and that's where we're failing now with the COVID vaccinations, is it costs a lot of money, whereas in the Salk case, we've leveraged the healthcare system so it's free because they're gonna come in anyway, and your baby just gets that vaccination. So that's, that's kind of a running example that is really useful to know because it checks all the boxes and as you know, it's scaled with great brilliance.

15. 1:20:49 – 1:29:26

    Unintended consequences in scaling: tipping rollout, network effects, and driver-trip incentives

    1. TFTim Ferriss1:20:49

       Could you expand on the unintended side effects, the spillage, maybe give some examples of where things have gone well or gone sideways?

    2. JLJohn List1:21:01

       Yeah, absolutely. So, so one thing that is invisible to the entrepreneur or invisible to the firm is an unintended consequence or what happens after I push my idea out there. Now, one example of an unintended consequence is what happened to us at Uber. So remember summer of 2017, we have this idea that we are going to add tipping to the Uber app, and in doing so, we beta test it. What we find is we go to a market and we allow 5% of the drivers to be able to receive tips. Okay, so what happens? They receive tips, they work more, and they make more money per hour. So it looks great. When 5% of the drivers have it, their wages go up and their labor supply goes up. Win, win, win. Now, when you roll tipping out to every driver in that market, guess what happens? They see tipping, they work more, but the market dynamics, because everyone has tipping, they undo the tipping effect. In fact, more drivers are working, but they're driving around with an empty car more often because so many drivers are driving that the extra tip effect is exactly offset.

    3. TFTim Ferriss1:22:49

       Huh.

    4. JLJohn List1:22:49

       So that's an unintended consequence now when we scaled it up. Drivers don't make any more per hour with tipping. They exactly offset it, and you didn't see that when only 5% of the drivers had tipping.

    5. TFTim Ferriss1:23:02

       Mm-hmm.

    6. JLJohn List1:23:02

       That's an example of an unintended consequence that caused all the good stuff of tipping to go away. Now, on the other side, sometimes you have good stuff. Think about Facebook. So Facebook, if only five people in the world have Facebook, it's pretty useless, right? I, I, it's not very valuable to me. But if every one of my friends and every one of their friends and every one of my family members is on Facebook, there's something called network externalities that as more and more people get it, Facebook becomes even more valuable. So you need to pinpoint, does your idea have these really cool network externalities in this case, or at scale, the good stuff becomes even better? These are things that I'm asking people to measure and take account of.

    7. TFTim Ferriss1:23:57

       Makes me think of Andy Grove at Intel, and I think he used to refer to it as paired incentives, but for any incentive, and I, I'm, I'm sure I'm getting the details wrong here, but he would, he would ask, like, what is the unintended side effect?

    8. JLJohn List1:24:11

       Yeah.

    9. TFTim Ferriss1:24:11

       Like, what is the sort of perverse behavior that you are, uh, and I mean perverse a- as in undesirable, uh, [laughs] behavior that you are also incentivizing that is not explicit, right?

    10. JLJohn List1:24:27

        Yeah, 100%.

    11. TFTim Ferriss1:24:28

        And you should measure both. Let me make this just purely self-indulgent for a second. So let me lay out a challenge, and you're the perfect person to ask. So I have seen this and heard of this across ridesharing apps, so I don't, I don't think it's unique to any one. But if dri- I have certainly run into this. If drivers are penalized for canceling trips, but when they accept a trip, they realize, "Oh, wait, that's a really short ride. I want an airport ride," or something like that, they will sometimes turn in the opposite direction and just drive away to force the rider to cancel.

    12. JLJohn List1:25:08

        Yeah.

    13. TFTim Ferriss1:25:08

        And then the rider gets penalized, and, uh, I'm wondering how, presented with that issue, how you'd think about or how you have thought about mitigating it or, or solving it.

    14. JLJohn List1:25:22

        When we did a deep dive into Uber wages across men and women- What we found is something really surprising. We found that men earned about 7% more per hour than women as an Uber driver. And that really surprised us because, you know, wages aren't negotiable. All men and women have the same rate card, and what a rate card means is you're paid the same time and distance. And women receive more in tips than men receive. So we were really surprised, and one reason why is because men strategically reject trips and women accept all trips.

    15. TFTim Ferriss1:26:14

        Huh.

    16. JLJohn List1:26:15

        So a bad trip on Lyft or Uber, y- you're right, o- one type of bad trip is I have to drive a half hour to pick the person up-

    17. TFTim Ferriss1:26:26

        Yeah

    18. JLJohn List1:26:26

        ... and then I only have a five-minute trip.

    19. TFTim Ferriss1:26:29

        Right.

    20. JLJohn List1:26:30

        One way to get around that problem is to pay the driver for time that they spend from when they get the dispatch to when they pick the customer up. We, we do that now at, at Lyft, and that's an important reason that you can lower rejection rates. Now, there are still cases of cancellations. What we do is there's a fine for, for drivers. In fact, there's a fine for, for riders who do this, too, right? There's a cancellation-

    21. TFTim Ferriss1:27:05

        Yeah

    22. JLJohn List1:27:05

        ... fee of $5. And I think using that mixture of actually paying drivers to go pick somebody up, it, and, and having the relevant fines, right, taking money away, clawing it back, is, is kinda two instruments that we find can be successful in lowering exactly what you're saying, 'cause it's irritating to sit out- outside in the cold. I'm in Chicago here, and it's freezing. I sit out on a corner waiting for a, a driver, and I see them going the wrong way because they don't wanna [laughs] pick me up.

    23. TFTim Ferriss1:27:39

        [laughs]

    24. JLJohn List1:27:40

        I'll feel a lot better if I know they're being penalized versus if they just go to a better trip, right? And you're right, the best trips are, "I'm gonna deadhead to the airport."

    25. TFTim Ferriss1:27:48

        And you also can't use your app while they're attempting to force you to cancel, right? So-

    26. JLJohn List1:27:54

        That's right

    27. TFTim Ferriss1:27:54

        ... I remember staying with a friend in Malibu once, and there's PCH, and it's a mess, right? And if the drivers are not paid for their transit time to the pickup, then more often than not, I mean, you would confirm the ride and then go back to doing what you were doing, knowing that it would be there in 12 minutes, and then check 10 minutes later and notice it's canceled. So you can solve that by paying them, the driver, by, for the transit time. What if it's a situation where it's, like, downtown Chicago or downtown Austin or wherever it is-

    28. JLJohn List1:28:26

        Mm-hmm

    29. TFTim Ferriss1:28:26

        ... and they're just looking for longer trips, so they start going in the opposite direction? How would you address that?

    30. JLJohn List1:28:32

        So they're not supposed to know that until you get in the car, right?
16. 1:29:26 – 2:13:45

    Four ‘after launch’ tools: scalable incentives, marginal thinking, optimal quitting, and culture building

    1. TFTim Ferriss1:29:26

       Yeah. So I'm gonna provide some foreshadowing for a question to come, just so it can slow bake.

    2. JLJohn List1:29:31

       Sure.

    3. TFTim Ferriss1:29:31

       So I wanna talk about the, the four secrets to high voltage after launch: scalable incentives, marginal thinking, optimal quitting, and building culture. So I'm gonna set that on just keep warm for now. I, I have to ask you, though, because you in, in a sense alluded to this, but we didn't get into the details. So [laughs] you received your PhD. You went then to sort of sell yourself on the academic job market, this is '96, applied to 150 jobs. Usually the hit rate is around 20, so one would expect around 30 interviews. You got one. So [laughs] there's that.

    4. JLJohn List1:30:08

       [laughs]

    5. TFTim Ferriss1:30:09

       And then you were able to sort of alchemize your career trajectory, and here you are, right?

    6. JLJohn List1:30:14

       Yeah.

    7. TFTim Ferriss1:30:14

       Obviously in a very different position. And my question is what did people miss about you, or what did you miss, or what mistake did you make, or what did you lack? Why did that happen?

    8. JLJohn List1:30:29

       Yeah, it's a really good question. I've been doing, uh, the post-op for over 25 years now.

    9. TFTim Ferriss1:30:36

       [laughs]

    10. JLJohn List1:30:37

        And, and I think it's sort of two parts. One part is academia is very hierarchical, and the way the funnel works is that the top schools, you know, Harvard, University of Chicago, Princeton, MIT, et cetera, w- we put out about 20 to 30 students per year on this market with PhDs. So we put out 25 students this year with a PhD from the University of Chicago. But we only hire back one or two. So the unwritten rule in academia is where you get your PhD, the ranking of that school will be much, much higher than you, where you get your first job. And then where you get tenure- Will be much, much lower ranked than where you get your first job

    11. TFTim Ferriss1:31:39

        Ooh, tough gig

    12. JLJohn List1:31:40

        That's just sort of how the funnel works. So when you start at the University of Wyoming, which is, I'm not sure what the ranking is, but it's certainly not in the top five schools. If you follow that rule, those top schools, they're not even gonna open my envelope. I, I didn't know this, but when I applied to the University of Chicago, I'm quite confident that my application envelope was never opened because they saw the return address was Laramie, Wyoming. Okay. So I don't think it was anything that I did wrong per se, although it is true that people were very, very skeptical of my field experiments in the early '90s. Everyone said, "You're an idiot for doing it. You know, we think you seem smart. Why are you throwing your career away?"

    13. TFTim Ferriss1:32:36

        [laughs]

    14. JLJohn List1:32:36

        "If you wanna do experiments, you do them in the lab. What is this field experiment stuff?" So you're right, I applied to 150 schools 'cause I wanted to be an academic. I didn't wanna be a truck driver. Nothing wrong with truck driver, but that was my outside option, let's be honest, right? If I don't get an academic job, I'm back in Madison, Wisconsin, driving truck. So I applied to 150 schools, and I still wear that today. I got 149 people who told me, "You're not even worthwhile talking to in an initial interview." And I saved every one of those envelopes that said, "Thanks, but no thanks."

    15. TFTim Ferriss1:33:16

        That's great.

    16. JLJohn List1:33:16

        And if I ever need motivation, I can look at that box of envelopes that says, "Go fly a kite List."

    17. TFTim Ferriss1:33:24

        Yeah. When you get your yacht, you can laminate them and put them in the floorboards.

    18. JLJohn List1:33:29

        [laughs] Yeah, yeah. You know, University of Central Florida worked, and it, it ends up working well I think too because what I see is at the University of Central Florida, I could take chances. And, and I can continue to take high variance plays. And what I mean by that is take risks in, in the research, where if I started at the University of Chicago, it might've been harder because I might've wanted to march to the same beat of every other drummer-

    19. TFTim Ferriss1:33:57

        That's true

    20. JLJohn List1:33:57

        ... around me. And, and I think U- you know, University of Wyoming as a grad student, University of Central Florida, th- these sort of relieved the social constraints of having to be like everyone else, and I could just be me. And I think in the end, it really helped a lot, and that was kinda the silver lining of my approach to academia, I think.

    21. TFTim Ferriss1:34:20

        I have to ask you, and this may be a dead end, and we can cut it if it is. This is gonna show you just how much I don't know, which is not gonna come as a surprise to anyone listening.

    22. JLJohn List1:34:30

        Mm.

    23. TFTim Ferriss1:34:30

        But, you know, when I, when I think about the cold reception that you received based on [laughs] the field studies and the sort of directionality of real world experimentation, people are like, "You seem smart. This is stupid. Why are you being stupid?" kind of reception, it reminds me of a, a documentary that I recently watched. It's a Nova documentary on fractals and Benoit Mandelbrot. And I'm wondering, because part of that documentary discussed the w- the wide spectrum of applications for fractals, lots of applications to physiology, cardiology. I mean, uh, extremely wide-ranging. Is there any application for fractal geometry or fractal mathematics to economics? Is there any intersection there?

    24. JLJohn List1:35:25

        I think there absolutely is, and I think as we move to higher order computing and as we move to a higher order understanding that humans have of how to apply mathematical principles to real world questions, I, I think some of the big breakthroughs will be in that area. And I think part of it is going to be how do you solve these problems and get a, what's called a closed form or an analytical solution rather than have something be, well, anything can happen. Because when you deal with humans – you know, the, the sciences are great because we have laws. And we can talk about these laws because they're physical constants. They apply everywhere that we at least understand they can exist, these laws apply. In economics, when you talk about predicting human behavior, we don't have quantitative laws. And, and what we have are things that on the one hand are kind of obvious, like our big law – one of our big laws is the law of demand. And basically what that says is when price goes up, quantity demanded goes down. Well, of course. If price goes up, you buy less. And then what we don't even have a physical or quantitative estimate because that's called an elasticity, but the elasticity depends on the good and the circumstance, et cetera, et cetera. So we always have to measure it. I think when you start to add fractal in o- in other aspects, it, it gives us a shot at least to try to be more scientific, and I think going down those dimensions can be useful. I can't imagine an application right now, but that's the fun part about the question, right, is that there are possibilities. 'Cause back when I started doing field experiments everyone said In part they're dumb because I'm going to a baseball card convention to gather data, and they said baseball card collectors are weird and-

    25. TFTim Ferriss1:37:53

        [laughs]

    26. JLJohn List1:37:53

        ... the results won't generalize. And, and that might be fine. It- they might generalize or they might not. But what I argued was the tool will generalize, and when we get to work with firms or governments or public education, we can do large scale field experiments there just like hard scientists, and then we can say something causal about the relationships that we observe.

    27. TFTim Ferriss1:38:19

        Yeah. Well, uh, and also if you're working... I mean, this is, obviously the reference here is within the halls of academia, but if you're working with private industry also, [laughs] like that weird group could actually generalize to a pretty larger, like a much larger-

    28. JLJohn List1:38:35

        Yeah

    29. TFTim Ferriss1:38:35

        ... subset of the population that is, like, equally weird as a standard deviation from whatever the hell normal is.

    30. JLJohn List1:38:40

        [laughs] Exactly.

Episode duration: 2:14:06