Jeff Weinstein: Why a burning problem beats clever ideas

1. 0:00 – 10:16

   Jeff’s background

   1. LRLenny Rachitsky0:00

      Watching you operate on Twitter, you're just breaking this wall between the PM and the customer.

   2. JWJeff Weinstein0:04

      The moment the customer felt compelled enough to go out of their way to talk about some problem, that's a unbelievable gift. I will leave a meeting to just get one message back to them. If you're text-message friendly with five or ten of those, you are going to have so much direct signal that is infectious.

   3. LRLenny Rachitsky0:22

      Many people told me they needed to ask you about picking metrics.

   4. JWJeff Weinstein0:24

      Well, what was the value that we're trying to produce for the customer and can we measure it from their perspective? And okay, how do you know you have product-market fit? Charts that showcase things are going up into the right on one hand, and then tweets on the other.

   5. LRLenny Rachitsky0:36

      You started at Stripe something called study groups.

   6. JWJeff Weinstein0:39

      We show up, four to eight people total, pretend to be some company with some outcome problem. Rule one is you do not work at Stripe, and rule two is we're not here to solve any problems. This is just about practicing empathy for the customer.

   7. NANarrator0:55

      (instrumental music) Today my guest is Jeff Weinstein. Over the course of his six-plus years at Stripe, Jeff was the product lead for Stripe's payment infrastructure teams where he helped scale Stripe payments to hundreds of billions of dollars in volume a year. He also led PMs and teams on a number of zero to one bets at Stripe, and most recently took on the scaling of Stripe Atlas, which as of the day this podcast launches, allows you to incorporate a new company in a single day, including handling 83b elections, incorporation documents, getting your EIN, share purchases, and all the things that used to take weeks or months before a company could begin operating. At this point, one in six new Delaware corporations are started on Stripe Atlas, which blows my mind. This episode ended up being the longest in my podcast history, because I wanted to basically do an archeology of an incredibly effective and admired product leader. We spent the entire conversation digging deep into the many skills that Jeff has built that enable him to consistently build successful and beloved products. We get into his go, go, go plus optimism long-term compounding philosophy of building products, how to think about and operationalize product craft and quality. He shares a popular program that he started at Stripe called Stripe Study Groups that I think you should steal. We also talk about how to effectively talk to customers, how to know if you have product-market fit for your new product, how to pick great metrics for your team, what he's learned about getting shit done at a big company, also advice that he's gotten from the founders of Stripe, and so much more. This episode is for anyone who's looking to level up their product building chops in every way. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It's the best way to avoid missing future episodes, and it helps the podcast tremendously. With that, I bring you Jeff Weinstein. Jeff, thank you so much for being here, and welcome to the podcast.

   8. JWJeff Weinstein2:50

      Thank you, Lenny of Lenny's Podcast.

   9. LRLenny Rachitsky2:52

      (laughs)

   10. JWJeff Weinstein2:52

       I was- I wasn't know- I- I knew what to expect, but it is fun to see the first name and the podcast all line up. I really appreciate you asking.

   11. LRLenny Rachitsky2:59

       So I wanted to start with a quote that I found from you that I think gives a little perspective into how you think and how you approach the world. So here's the quote, "Very frequently I would do poorly on tests in school, and then the professor would say very reasonably, 'Hey, I think you should bump down a level to the previous semester's pace.'" And you said, uh, "I actually know that. That's why I'm in this class. I want to be in the class that I'm potentially the worst at." This isn't how most people think. This isn't how most people operate. Usually people want to get good grades, they want to be at the top of their class. Clearly you have a different approach and a different mindset. Where did this come from for you, and how did this shape the way you think about product and- and the work that you do?

   12. JWJeff Weinstein3:37

       Well some of it was just the fact that I wasn't particularly good at the class and had to, uh, rationalize it for myself in some form.

   13. LRLenny Rachitsky3:43

       (laughs)

   14. JWJeff Weinstein3:43

       So, uh, in retrospect that sounds kind of highfalutin, but at the time I just wasn't particularly good at the classes I was in. But, I think it comes from growing up I went to a pretty hippy-dippy K through 12 school in Baltimore, Maryland, where we were really asked to think about why we were in school and to pick any of the courses that, uh, were of interest to us outside of AP programs or grades or any particular requirements. You really got to choose your own path. And I recall one particular class in high school, uh, which was sort of somewhat a science class, but it was called The History of Science. And we actually walked through and studied all of the, at the time, best understood ways the world worked in science, but then later were turned out to be wrong, right? In the 1500s we believed X, Y, and Z. In the 1600s we've A, B, and C, just very confidently in 1500s we thought something, and then in the 1600s they thought something very different. And so this class was quite impactful on me where we spent an entire year studying things that are not true, and it was fascinating. And that particular teacher employed another trick, uh, on us during that class which was, um, he took the tuition fee of our school and divided it by the number of hours and wrote- wrote the cost on a ticket, and- d- and then handed us in the beginning of the year tickets for every single one of the classes. And he would stand at the door, and you would have to give him a ticket at the end of the class if you thought it was worth the- worth- like worth it. And just like that practice of deep intellectual understanding of how people evaluated something at the time, and choosing for yourself to spend the time on it by just like the physical act of handing that ticket to the teacher, that really clicked for me. And so when I got to college and it was a real co- real college, university, people were s- coming from often quite more rigorous backgrounds of things that were true, uh, I was a bit unprepared. And I remember actually like taking a microeconomics class that was quite advanced and had a close friend, and we studied exactly the same information. We sat, looked at the same cheat sheets. We practiced the same quizzes. We read the same books, and he got the top grade in the class, and I got the bottom. And-That's the, from where that quote came from, where the professor said, "I think you should bump, potentially bump down." I was like, "I, I already know that stuff. Like, I'm interested in this topic. I'm gonna try to improve, but look, just because I'm significantly worse than the other kids in the class, that has, like, little to do with if I should leave." And he was particularly cool with it. (instrumental music)

   15. LRLenny Rachitsky6:32

       This episode is brought to you by Pendo, the only all-in-one product experience platform for any type of application. Tired of bouncing around multiple tools to uncover what's really happening inside your product? With all the tools you need in one simple-to-use platform, Pendo makes it easy to answer critical questions about how users are engaging with your product and then turn those insights into action all so you can get your users to do what you actually want them to do. First, Pendo is built around product analytics, seeing what your users are actually doing in your apps so that you can optimize their experience. Next, Pendo lets you deploy in-app guides that lead users through the actions that matter most. Then Pendo integrates user feedback so that you can capture and analyze what people actually want and the new thing in Pendo, session replays, a very cool way to visualize user sessions. I am not surprised at all that over 10,000 companies use it today. Visit pendo.io/lenny to create your free Pendo account today and start building better experiences across every corner of your product. PS, you want to take your product-led know-how a step further? Check out Pendo's lineup of free certification courses led by top product experts and designed to help you grow and advance in your career. Learn more and experience the power of the Pendo platform today at pendo.io/lenny.

   16. NANarrator7:53

       (instrumental music) Pendo.

   17. LRLenny Rachitsky7:56

       Today's episode is brought to you by Cycle, the AI-powered feedback platform for product teams. Is your customer feedback a tangled mess of Slack threads, survey responses, and overflowing inboxes? Wish that you could know what your customers really need? Cycle unifies all of your customer interactions from support chats, to user research, Gong calls, and app store reviews into one neat collaborative space. Cycle's AI then extracts actionable insights on autopilot. Cycle will learn what you're building so that it can label incoming feedback automatically. That means you'll get a full voice of customer report without manually triaging feedback. Then simply use Cycle Ask to dig deeper into any topic and generate custom AI-generated summaries across your entire feedback repository. What makes Cycle different is the way that it lets you close feedback loops in each release. Feedback is not used just as a way to prioritize what to build but also as a tool that creates trust with all stakeholders. Sign up for a free Cycle trial today at cycle.app/lenny and put your feedback on autopilot. That's C-Y-C-L-E dot app slash Lenny. It's really interesting that you went to this liberal artsy HEP D.P. school as you described, which a lot of people bash on. Like, why spend all this college money and time on a liberal artsy kind of education? And, and you end up being really successful in a very technical, highly analytical, data-driven company, and it's interesting that that experience still helped you succeed in this other path.

   18. JWJeff Weinstein9:27

       It, it did take a change though, because I started in... English was my selected major in school, but then I kept playing with computers and I kept liking math, and I looked at the roles of the, you know, the backgrounds of the people who were running the types of companies I loved. At the time Facebook was getting super, phenomenally successful. Apple was already on their, on their up and up again, and all, everyone, everyone who was leading those places had technical backgrounds, and I liked computers a lot so I added computer science as a en- as an engineering degree midway through school. So, ended up having to take those same... Had to take the real science classes with the pre-med kids and the rest of it, and I did similarly poorly. But I, I did end, end, uh, up with a computer science degree and, and a liberal arts degree. So it was, it was a journey, but I had to make a, a pretty big switch in the middle to, like, get on that path.

2. 10:16 – 15:38

   The “go, go, go ASAP + optimistic, long-term compounding” approach

   1. JWJeff Weinstein10:16

   2. LRLenny Rachitsky10:16

      I asked you if there was, uh, one thing that you'd love to get across in this podcast. I asked you what would it be, and here's what you said. Here's the first item on this list that you sent me, quoting, "Go, go, go ASAP plus optimistic, long-term compounding approach." Can you just talk about-

   3. JWJeff Weinstein10:35

      Yeah. (laughs)

   4. LRLenny Rachitsky10:35

      ... what you mean by that?

   5. JWJeff Weinstein10:36

      Yeah. There's two things going on here. So... I s- I sort of see the world as immediately we need, we have just such opportunity to take action in front of us. We can be optimistic and go, go, go as soon as possible. I think that a lot of life is h- you know, you get as much furniture as youve as ho- as you have room in the house. We will do the work the night before it's due. So let's just make it due tomorrow. Can we turn tomorrow into today? And so just optimistically seeing if we can just inject energy to go, go, go has produced surprising results and I think it ignites in other people that same interest, and then it kind of feeds off each other. And that's, I think, really in my bones from growing up. And then I added over time, had to learn this longer term compounding, more strategic mindset where some of the things we want to accomplish, be it at my startups in the past or, or at Stripe, they can't be solved in an afternoon. They're going to require layers of infrastructure and services and applications and UI and partnerships that really look like that kind of iceberg drawing you see where you just see the top, but then there's the whole thing underneath. And I've had to learn over time to pair my instinct of like, let's get it done today, let's move forward, let's, let's see what we can get done, let's make some mistakes, let's try it out, with where are we going? What needs to be true over time? Where can we always invest? What, what, what will we sort of never regret spending time in? We'll never regret spending time making the latency of our payments APIs at Stripe faster. We'll never regret, uh, making it more reliable to send 83 (b) election mails to the IRS. Like, we will never regret those things. So let's just always compound, uh...... those capabilities over time. Then the trick is, like, how do you mix this go, go, go attitude with a long-term compounding? And that's something I still struggle with, but I try to purposely balance it more than I used to.

   6. LRLenny Rachitsky12:44

      It's such a good way of summarizing just how to be successful in a lot of things. Go, go, go, ASAP, stay optimistic, focus on long-term compounding growth. Is there an example of that in action? I, I, I know we're gonna talk about Atlas a bunch, but I guess is there an es- example of that working for you as... or a product you worked on?

   7. JWJeff Weinstein13:03

      I work at Stripe, which is a, a, an infrastructure company. We build things that help businesses do online commerce in various forms, and I've had a few roles at Stripe. Um, in our beautiful office here in South San Francisco, one of which was, was the product person that helped us decide how we were going to go global and how we were gonna offer multiple ways for people to pay. It turns out there's more than just credit cards in the world. There's small hundreds of ways that people will naturally want to pay for things online, and as a business you're gonna, of course, just want to accept whatever it is people want to pay with. And for the first seven or eight years of Stripe, well prior to my, my being here, you know, the, the incremental country ads and the incremental new payment methods was relatively flat over time, and that was surprising to all of us who worked here because the world wanted it and we had a lot of people working on it, and we were working very hard. But it wasn't producing returns in the way that we wanted, and so actually the optimistic, as soon as possible, go, go attitude was not working, right? No matter how much energy we poured into building Thailand payment methods or UK bank transfers or, uh, you know, an in-person payment system in Latin America, we just like couldn't rack up points and get it done. So we had to step back and say, "Well, what is the world gonna look like in 10 years? What was going to need to be true and how can we start to go, go there now?" Which meant going a lot slower, building internal platforms, sending people around the world to start to build these payment methods, up- uproot their lives, pay for their apartments, get them on airplanes, start using these payment methods actually in the world. And so it really... like, our line flat-lined for a while, and we're like, "Okay, is this strategy working? Is it not working?" But then over time we start to see it switch to nonlinear again and go from whatever it was, 10 payment methods at the time, and now Stripe accepts over a hundred, so we got to, like, 50 really quickly and then kind of like skyrocketed to 100. And I remember there being a meeting where we said, "Well, maybe we should just, like, lock everyone in the basement and see if we can get from 10 to 50." That would be, you know, the intensity startup, go, go, go attitude. But we looked at the individual components of what it meant to get this done and how, how long we wanted this to be true for, and we had to go a lot slower. So that was like a very formative decision where you had to mix the go, go, go with the long-term compounding.

3. 15:38 – 24:15

   The importance of craft and quality

   1. JWJeff Weinstein15:38

   2. LRLenny Rachitsky15:39

      Awesome. Okay, so let's start to delve deeper into some of the specific skills that I hear you're incredible at and that I've seen you be incredible at on Twitter and, and LinkedIn and things like that. So the first is craft, craft and quality. I'm told by many people that you have a very strong, uh, obsession with craft and user experience and quality. And even more so I'm told that you teach people at Stripe how to be obsessed with craft and quality and the user experience in a very systematic way. I feel like that's something a lot of people are starting to realize is really important and/or are trying to get better at either personally or at their company. So first of all, let me just ask why is, why is this so important to you? Why do you think craft and experience and quality is so important? Why do you put so much emphasis on it yourself?

   3. JWJeff Weinstein16:25

      I think I'm really working backwards from failures in the past (laughs) and avoiding them, and so maybe just a quick story. I had this... I started a company several years ago based on just a personal pet peeve of mine, uh, which was I was a SQL analyst. Many of you listening might have written SQL in the past. Maybe the robots write the SQL for you now, but, uh, you still need to write a little SQL yourself. And every single time I wrote a SQL query, I wanted to run the same subsequent analysis. I wanted great version control. I wanted all the cool stuff that GitHub had for code but in a, in, in... but for writing data code. And so my friends and I built a little Python tool which, you know, basically let you run R-style queries. It let you draw, uh, charts and pivot tables quickly and sharing all, all... sort of all this mo- modern SaaS application stuff but applied to SQL a few years ago. And we turned that into a company. We got some progress, but then there was this moment one day where we had maybe a couple hundred customers and we had an error where we basically by accidentally shut down the service and it was sort of bricked for 10 or 20 minutes. And at the time we were all kind of hustling in our little dinky office to get the application back online and we really kind of were proud of ourselves about how reasonably quickly we did and people went back to using it, it wasn't a super long outage and we didn't lose any data. We were just like kind of high-fiving each other and we went about our day. And about a year later I realized that that was a... I missed a huge moment that I should have pounced on, which is that during those 20 minutes, our customers weren't furious. They weren't emailing us like crazy, they weren't texting us, they weren't trying to find us on Google Maps and knock on our door and say, "Hey, I need this thing back online immediately." We heard a couple comments from them. It was little murmurs.And I didn't realize at the time, like, that was the signal that I did not have- like, we did not have product market fit. And I ended up wasting many more years on that project, and wasting is a sort of big word. We built amazing software. People liked it. We were able to sell the company. It- it- it helped, it helped many of us learn how to really build software.

   4. LRLenny Rachitsky18:57

      Mm-hmm.

   5. JWJeff Weinstein18:58

      But I'm- I'm really trying to avoid that situation again. And I think craft is kind of a dessert that you get after the meal of does your thing solve a real problem in the world, and are people clamoring, needing it badly. And that's really my obsession is in finding problems in which people will pause their entire day to solve. They will leap through the computer to- to be like, "Oh my God, I have that problem. Do you have a solution?" And if you, you know, focus on that style of product development, and we can get into just ho- h- how do you kind of, like, listen for that and then turn that into product. Later, you might get the opportunity to really provide craft and beauty and touches and moments and delight, but certainly there is no amount of craft, there is no amount of beauty, there is no amount of delight or touches you can add to a thing that will solve the problem we had at our- at our startup, which is that people didn't really need this. Um, and that's sort of like the biggest error is like picking something in which people don't really need it and then going through these practices of trying to like make it great when maybe it shouldn't exist in the first place.

   6. LRLenny Rachitsky20:16

      I think a lot of people see all these tweets and messages about just, like, obsess with the craft of what you're building, and you can easily lose sight of nobody- nobody even cares about what you're building. It could be the most incredible experience ever designed, but if it's not something anyone ever wants, it doesn't really matter.

   7. JWJeff Weinstein20:34

      People don't really get out of bed for their second problem, right? They get out of bed for their first problem. And you have to carefully listen and not pitch your customer or your prospective customer as you're trying to- trying to figure this out. And I think this advice, not even advice, just style applies to small companies, big companies, anyone, which is people don't want to be pitched. You know, I- I- I'm sometimes on a UXR call with a very well-meaning person or a founder who has a new product that they want advice on or they want, uh, to find customers and the first thing that they start talking about when we get on the call is, "Hi, I'm the CEO of X, Y, and Z company. We do one, two, and three. I want to show you a demo." It's like, ho- hold up. (laughs) Uh, I'm- I'm a customer. I have... Like, what a wasted opportunity you- you've kind of- you- you've just done here. I have so many problems. Ho- you're sort of guessing ahead of time what is my top problem, and now that you've anchored and limited to the pitch, you're- you're going to miss, you poten- very likely going to miss the burning problem that they have on the top of their- on the, like, on the top of their mind and they're, you know, it's not the customer's job to interrupt you and say, "Hey, could you stop your pitch? Like, I want to tell you about my- my top problem." And you can sort of see this downstream which then companies who launch things, put craft, uh, have a- have a great launch, raise money are then, like, later gone. Right? And it's because they built something that wasn't solving a burning need. And I think you can kind of stem it all the way back to they were just pitching rather than sitting in silence and just waiting for their customer to just open their heart out about what's the most burning thing. And sometimes I'll- I'll prompt our customers. I'll just, I'll say, "Hey, do you mind just opening up your email? Uh, like, what- what's in there?" Or, "If you weren't talking to me right now, what would you be working on?" Or, "Hey, last week, wha- what- what grinded your gears? What, like, what are you not looking forward to?" Or magic wand, like, "What do you wish we- you could just, like, have off your plate immediately? Forget- forget Stripe, forget about just thing we're working on. Just in your whole life, like, what is it that you do not want to be doing or a massive opportunity you wish was just one door away?" And it's a little awkward 'cause they don't know why you're asking that kind of question, but then you just sit there in silence and for the most part, if you have amazing customers who are smart and ambitious and are trying to build their own business, they're going to want to offload their hardest thing to somebody else if you've earned trust along the way, and certainly not pitching is a great way of earning trust. You're just listening. We- we've- we've learned a lot at Stripe from that and that's where you can start to find adjacencies, right? Well, hey, I don't know if Stripe could, you know, really help me with this, but wouldn't it be cool if- if, uh, we could identify if the person signing up on our site, like, wasn't fraudulent or is who they say they are or isn't a bot or is- are they really a dog on the internet? We're like, huh, we keep hearing this as the major complaint from all of our customers and, but, like, we're not an identity company.

   8. LRLenny Rachitsky23:50

      Mm-hmm.

   9. JWJeff Weinstein23:50

      It's like, well, I guess now we are because all of our customers who are trying to build their business, like, all have this problem and so via silence you can just create your roadmap pretty quickly and drop a lot of the long UXR, long survey, long build cycle approach.

4. 24:15 – 28:57

   Effective customer communication strategies

   1. LRLenny Rachitsky24:15

      And so I love where we're going with this. This is where I was gonna actually talk about next is just, uh, your ability to talk to customers and use research. It feels very unique. Watching you operate on Twitter, you're just, like, sharing your email constantly, uh, getting on calls with people constantly. There's this, like, you're, like, breaking this wall between that a lot of people imagine there is between the PM and the lead of a team and the customer. Like, you're not relying on user research. You're not waiting for someone else to do this work for you. Uh, talk about just...... why and how you do this, 'cause I think there's so much to learn from just how you operate in finding opportunities, picking problems to solve by talking to customers.

   2. JWJeff Weinstein24:53

      It's where the business comes from, is customers. You know, it's not a long-shot, um, hypothesis about why to talk to them. It's like, th- if they love your stuff, they will tell their friends and pay fair prices for the product. We're so fortunate through the internet that people kind of announce themselves as having, as being interested in a topic. Sometimes they are interested in it by posting on Reddit a long thread or a screenshot of a customer service interaction that bugged them or hopeful that from their dorm room in country X, someone else in the world has solved the same problem. And the internet has given us all this absolutely magical forum that you can just yell out the window, and then billions of people could just, like, li- c- could actually listen to what you said out the window. Now, that's not all the time true for all people and all situations, but dramatically more true now than ever before in human history. And so the fact that people wouldn't be listening to their customers and, and really jumping through the computer to talk to them surprises me. It's like they have mu- like, I'm, I always sometimes think, do they have some other strategy? Are, are they so confident in what they're building that they don't need to hear directly from the people who will be using it? And I think some of it is my own fear that I'm going to make the same mistake I've made in the past, which is build something that people like, they're using, but isn't solving a burning enough problem such that they're going to stop their day, they're going to tell their friends, we're gonna be able to sustain the company economically over a long period of time. And that really just comes from hearing people's most burning problems. And then the jumping through the f- the, the, like, just jumping through the computer and talking to them takes a little bit of, of nervous gum- gumption, y- you know, kind of like walking up to a person at a bar or co- cocktail party and saying, "Hi, my name is Jeff." It, it can be a little awkward, but then you get it, it's kind of a rush once it starts to work. And once you get the iterative loop that they're excited to talk to you, they have problems, they see you as a trusted, not salesy, not pitching, not narrow-minded to just my, my product and my position, but I'm here to hear about your wh- your whole pro- all of your problems and see where we can help, not promising we can help solve everything, but let's listen, that is infectious, both between the customers and internally. And so I'm able to bring more people into that practice at Stripe. I'm able to quickly grab an engineer and hop on a call. I can forward a, a message over to a Slack group and they know that because the customer's speaking directly, it somewhat trumps everything that's happening during the day. You know, we, we could go to a meeting where we're gonna guess what's going t- the customer wants, or we could talk to them directly. Uh, and you have to use a little bit of art to decide which customers you want to listen the closest to, but even at Stripe's scale where we're dealing with many millions of businesses and many hundreds of millions of consumers on the other end of those businesses, you can pattern-match relatively quickly what are the styles of customers that represent where the world is going, the most ambitious, the most technical, the fastest growing, the most detailed, and you don't need 10,000 of those people to talk to. If you're text-message friendly with five or ten of those, you are going to have so much direct signal about where to go that you kind of forget how you did it in the past.

5. 28:57 – 33:19

   The importance of speed in customer interactions

   1. JWJeff Weinstein28:57

   2. LRLenny Rachitsky28:57

      I love these. So I'd love to learn more about these tactics that you've found helpful. So you've shared this idea of silence, talking to someone and just being silent. And I forget the phrase used, but just, like, ideas emerge. Like, your whole road map can emerge from that silence. So let me share a few of the things that you've shared so far, and I'm curious what else. Two is tweeting. Just like, "Hey, I'm looking to improve Atlas right now. What bugs do you have? Here's my email address." Uh, you talked about text messaging. Just like, "Hey, can I get your number and I'll just text you when I have questions?" If there's anything else to add to these, that'd be awesome, and then what else have you found just, like, ways to actually get to a customer and find opportunities that are important?

   3. JWJeff Weinstein29:34

      Speed is an important one, which is redu- just reducing the time between the moment a customer felt compelled enough to go out of their way to talk about some problem. They're busy. There's snacks to eat. They have families. (laughs) They have other things to do. There's a lot going on in the world, a lot. Your dumb product, it's amazing that they would spend any of their time discussing it at all. I mean, most of the time if you don't like something, you just move along. You just apathetically, silently move forward in the world. And so the fact that someone took their finite time to succinctly, with curiosity, communicate to the world or you about your product, that's a unbelievable gift that should be, you know, P0 alert-level intensity. And so I will leave a meeting, I will ch- change what I'm doing to just get one message back to them, even if it's, "Hey, I got this. I'm about to go to dinner."... "Can I hit you up tomorrow?" They're like, "Oh, yeah. Thank you. Awesome." Like, "I can't even believe you responded," that puts us in the camp of on the right trajectory, where they're going to feel that they have a almost secret portal between this big brand of a company and another human who's just actually curious what's going on. That's night, night and day. And it's also i- like fun fact, tip, when people are really hot on an issue and it could blow up on social or they're gonna start becoming a detractor, we make mistakes, we have SLA breaches, we have errors, you're gonna wanna get on that stuff fast. And m- in those situations, I... My bar for where, like for how we will respond to those folks is not to just sort of solve the problem but is to turn them into a promoter. And most of the time we're able to, even if there was a pretty relevant issue. I remember one time in Atlas, um, (laughs) we had this bug in which for a handful of our legal docs, they were handing out, let's say, 25 shares rather than 25% of the shares. We dropped the percentage sign. And thankfully, a, a founder noticed this in the docs and, and tweeted about it. And I was like, my heart like, uh, like paused, you know? And I was like, "Oh, no. This could be a ser- really serious issue." And we were able to fix it and regenerate the documents relatively quickly for everyone that was impacted. And I was thinking to myself, "Wow, we really, like, let this person down." We have one job to do, which is to get their company right, and we didn't. And that person was incredibly gracious about the situation and said, "Any time you want me to just read your docs, I'd be happy to. I have a law degree. I care about this topic. I wanna brainstorm with you about ways that Atlas could do more document creation." It was like, "Wow, uh, I can't believe I was in this sort of defense position, and now we've gained a friend in the world who can be eyes and ears and brainstorm with us." And we, the team maintains a really close relationship with that person, as I do with many, many, many founders who use Atlas. And so just, again, we, we're so quick to put the outside world in this other camp where we need to touch it with kid gloves and treat it as a big blob and a, and a cohort and a statistic, when it is a human with a problem who likes snacks who is busy. And it's fun to do, and it turns out to be an incredibly easy, fast way to figure out what to build.

6. 33:19 – 36:53

   Narrowing your focus

   1. JWJeff Weinstein33:19

   2. LRLenny Rachitsky33:19

      There's an, a bunch of stuff I wanna follow up here. One is, people may be hearing this and like, "Oh, my God. I'd be overwhelmed with feedback and people to fix problems for and bugs and texting of people." They're just like, um, there's so many people that potentially, uh, I'd have to be interacting with. How do you, how do you try to narrow that down? Like, I know Atlas is a very, uh, highly adopted product. There's a ton of people. There's probably bugs often, people sending you feedback. How do you pick the people to zero in on?

   3. JWJeff Weinstein33:46

      Uh, I think there's, uh, two parts to this. One, what a problem to have. (laughs) Of all the problems in the world, I'm overwhelmed by customer interest is on the, you know, the top list of problems I at, I want to have. I think most entrepreneurs would purchase that problem as a state. And so if you're in it, I think take a deep breath and look at the sun- sunrise or sunset and just enjoy the fact that you've built something that is meaningful enough that people would spend their time. Again, there are amazing snacks (laughs) , uh, that they could be spending their time on otherwise.

   4. LRLenny Rachitsky34:24

      (laughs)

   5. JWJeff Weinstein34:25

      So that's a, that's a huge deal. And then two, you need, you need the, the art... There's an art and science to picking where to go deep. You know, I will happily respond to folks with, "I really appreciate that. Do you mind sending me a screenshot or, uh, a video?" And some of them don't. Okay, that's fine. You kind of get a sense of quickly looking at how they wrote about it or a pattern, just pattern matching, how detailed they were or how much they seem that they want to engage and kinda, kinda balance it that way. Otherwise, you know, you can, you can tell folks, "This is really awesome. Do you mind sending me an email with three to five bullet points about the details of how you got to this issue?" And that's not a 30-minute meeting. You don't need to blo- blow up your whole life to get on the phone with them. It's not a huge homework assignment to them, but it's enough structure that it will self-select those who want to go deep. And then from there, you really do owe it to them to, to follow up if they're going to be that detailed. And at that point, you really have a sort of product friend for life, and you'll kinda continuously go back to them. So, y- you know, you, you, you have to kind of tune, tune it. I also will bound some of these efforts, so I'll just completely make up one day a program. So, "Hi, uh, Stripe's doing (laughs) a, um, bug finder program." I made that up as I was driving to work one morning, uh, tweeted it, was curious if anyone would send anything. We'll be taking videos for the next three days to go, you know, super deep.And people are like, "Oh, my goodness. I would love to be part of the Stripe bug finder program! Like, may I apply?" I'm like, "Uh, you know, it's very selective. Uh, you know. (laughs) Of, of course, but yes, of course, for you." And so just you're giving people permission through a pro- program, even if it is deeply arbitrary, I find helps with the bounding it. And then later I can follow up with the world, "Hey, uh, we ran our first, uh, program. We got 65 videos (laughs) , which we did. Uh, we found dozens of issues, which we did. Um, it was an incredibly valuable use of, of, of our time, and it, it really came from just..."... a, a single tweet.

   6. LRLenny Rachitsky36:52

      Mm.

7. 36:53 – 40:24

   Why you should pay attention only to paying-customer feedback

   1. LRLenny Rachitsky36:53

      Something I saw you mention somewhere is that you only pay attention to feedback from people that are paying customers, and ignore everything else. Can you talk about that?

   2. JWJeff Weinstein37:00

      People who build things for people tend to be empathetic and interested and curious folks who have friends, and then those friends wanna use your stuff because they know you and they like you, and they'll have good feedback, but you need to figure out, is that actually your customer or is that your friend trying it? Who is actually your target customer exactly? Not a company or a segment, you know, not digital natives that are X big. I'm talking about Sarah in this department who has these tabs open and just faced this problem and needs to solve it by 4:00 PM. Like, that level of specificity. If that's your customer, and I'm talking mostly about B2B, which is where I spend a lot of my time as... The s- social network B2C stuff I, I have less intuition on, they are very willing to exchange money for solving their problems. Incredibly so. Oftentimes they'll, if you listen with enough silence, they might even say, "I'll pay you money to solve blah," uh, if you, if you sit there long enough in silence. And so you could listen to your friendly friend who you got to p- play with your beta version and they'll say they liked it and they will click around, but that is extremely different from Sarah who has the actual problem and is willing to pay. And it is so tempting to go down the first path with the friend group that I sort of needed a rule which is like, okay, rather than... I was like, well, don't pay attention to that quite as much. You know, really pay attention to, to like who is your target customer and are you in a fast iteration cycle w- for them and are they telling their friends? But that wasn't enough because people are so drawn, myself included, drawn to the, the friendliness that I just said a rule, which is we discount all of that feedback from our friends to zero. Like, that is not of interest to us. It doesn't... We don't even write it down. (laughs) It's not par- it's just not part of what we're talking about at all. We are only interested in Sarah, our target customer, and can we get her to solve whatever the problem is as quickly as possible, as jankily as possible, and go from there? And it's a for- and then the paying part is a forcing function because, again, even with Sarah, because you're paying attention to her and solving her problem a bit, she'll say, of course, "Oh, this is great. I want feature X, Y, and Z. Can it do one, two, and three?" Naturally she would say that. But if you said, "And by the way, uh, this thing is $10,000, um, feel... We'll happily refund your money 100% the second you don't like it," she might say, "Whoa, whoa (laughs) . Wa- wait a second. I, uh, yeah, I, I like this thing, but not, not $10,000. For $10,000 it would need to solve X." And you're like, "Oh, there it is," right? The- that was actually what the thing that was of value and we were all kind of dancing around this its usefulness, its useful, we're on the right track. And because I've fallen down that path myself and so many times, I just set a rule.

8. 40:24 – 45:33

   Practicing silence when communicating

   1. LRLenny Rachitsky40:24

      This is great. I was actually gonna ask, this technique of using silence, to help people actually practice it, you just shared this kind of quote of, "For you, you to pay $10,000 it would have to do X." That's a really cool line to use. Is there any other advice there of just how to practice this idea of creating silence to help a potential customer share what they actually need?

   2. JWJeff Weinstein40:45

      I encourage people to take an improv training, improv training class, uh, or two, just to get people out of their s- skin a bit. By the way, this advice applies to big companies also, not just small ones. And we, we have issues, and I hear other larger companies have issues, where a big company will say, "Hey. We, we're an enterprise and we're gonna pay you this big contract. All you need to do is build features one, two, three, four, five, six, seven, eight, nine and go through security steps A, B, C, D, E, F, G," and the siren song of the big three-year contract of your first enterprise buyer, even when you're already big, it's, there's always a, a s- you know, it can go from Fortune 1,000 to Fortune 500 to Fortune 10 to Fortune 1, you know, th- there's always more. And a l- there are so many stories of the rug being pulled in the middle and they never, they never actually use it and the contract never lands, and same in partnerships. And, and so I would say the same thing which is like, if we're super serious about this, send us a million dollars. Like, wi- wire us a million dollars, we'll happily wire it back anytime you need it. But let's actually put some stake in the ground about the value. And I find that in the cases in which they're like, "Whoa, wait a second (laughs) . No. Uh, absolutely not. Uh, we're... We can't commit here," that's really good signal because you're about to spend a l- a huge amount of time building something that might not ever be used, and that's the majority case I hear, or you might get the opposite, which is, "Sounds great. Let's make sure our teams are trained on it faster. Uh, how can we get you to our office to explain it? We need it to really work with this system we haven't talked about yet, and now that we're paying, we want it even faster." And so I find that just it puts a fork in the road towards faster success or avoiding the non-success case. And even at Stripe's scale, I've heard our founders somewhat push this, this methodology on us, where you, one, from the outside might think, well, it's a 8,000-person company.... sure, sure, surely they have just regular ways of, of, of building things for larger customers and, and, and like we too need these style of commitments if we're going to go off roadmap.

   3. LRLenny Rachitsky43:04

      So big lesson you're sharing here is essentially make sure people are ready to pay for something that they're asking for. That is the ultimate sign that they actually want it.

   4. JWJeff Weinstein43:13

      And by the way, ready to pay is different than paying.

   5. LRLenny Rachitsky43:15

      Hmm.

   6. JWJeff Weinstein43:16

      Significantly different.

   7. LRLenny Rachitsky43:17

      Hmm.

   8. JWJeff Weinstein43:18

      Significantly different. I've also thought ab- of, of, well, as long as they're ready to pay and they said they would pay and they look at the contract, we should feel good. That is not the same as paying. Paying is an independent group of people saying my problem is burning enough that I'm willing to exchange something I have that has value for the promise of what you're going to do and now it's like a, a real commitment. That is extremely different than ready to. So I will often be on the phone or video, whatever, with a founder and I will have them practice paying, like have them practice charging me. I'll just say, "Hey, I'm just a friend. I'm trying to help you out." Uh, it's a little, it's a little bit self-serving 'cause I work at Stripe and I want to get feedback on our products, but I'll say, "Feel free to sign up for any invoicing or payment service, I don't really care which one, you're welcome to choose Stripe if it's easy enough for you, I'd love to hear your two cents, and send me an invoice or a payment link for a dollar right now. Like, right now. That way you don't, when it comes time to actually charge your first customer, it won't be your first time. It won't feel weird. You have already done this. You already have a dollar in your account. It'll just, you already have your logo on the top right of the invoice. It will feel great." And (laughs) I, I probably could go through my email inbox and just see $1 receipts to random people, uh, because I'm just so convinced that the difference between paying and, like, willing, ready, thought, I thought they would pay is, you know, ni- night and day different.

   9. LRLenny Rachitsky44:57

      Yeah. And there's this term willingness to pay, uh, that I feel like should change, as you're saying, to just cross out willingness to and just-

   10. JWJeff Weinstein45:05

       P- paying.

   11. LRLenny Rachitsky45:05

       ... paying, exactly.

   12. JWJeff Weinstein45:06

       Yeah. No, yeah. And we can refund the money. There's no, like there's no issue at all. Yeah. Other, just other tricks on how to get people going is ask them about their regular life as a human. You know, what did you do yesterday? And they'll be like, "Oh, at work?" I'm like, "Or, or not, whate- whatever." And people just, they'll start to spill. They'll just start to spill and eventually you'll get to their biggest problem pr- pretty quickly.

9. 45:33 – 54:08

   The role of metrics in product success

   1. LRLenny Rachitsky45:33

      Let's talk about metrics, kind of going in a different direction. Uh, many people told me I need to ask you about picking metrics and the importance of metrics and how you think about metrics, so let me just maybe start with a question of just why do you think picking the right metric and why are metrics so important in building successful products?

   2. JWJeff Weinstein45:52

      I somewhat walk around with the belief that, uh, the product manager's responsibility is to produce product market fit. And, okay, how do you know you have product market fit? Charts that showcase things are going up and to the right on one hand, and then tweets on the other. So metrics like, you know, quantitative and qualitative and I really see them as deep, deep siblings and equals. Y- you really need both. It's not, oh, OKRs versus something. Like there are some things you want the texture of the person on, on video showing how complicated a thing was, and then also we're trying to make an economically viable system that we can run at large scale and you can't keep all that stuff in your head and need to me- need to measure it. And so I think metrics at their best are a numerical representation of the value we're pr- providing for the cu- for the customer. One could measure anything. You could just like start counting events and log, log lines. We've made it incredibly cheap to count stuff. And so now we have this big privilege of choosing what to measure and I really just try to map it all the way back to, well, what was the actually the value that we're trying to produce for the customer and can we measure it from their perspective? Whereas I think it's natural both because we, when you work inside of a company, you're just thinking internally, but also the way that the metrics are collected inside your application to be more internal oriented. How many people logged in? Okay, that's a fine measure, but how many people accomplished what they were trying to do after they logged in is not just necessarily sitting there as a single event in your database. You have to think about it a bit. Another reason why I spend a lot of time cra- crafting a small number of these metrics is they force trade-offs and decisions. So we could all sit around all day and say, "Hey, I heard all these customer problems, we should build X, Y, and Z," and another person could absolutely reasonably say, "Well, I heard from these customers we should build one, two, and three," and they're all true. We could have a lot of success in both, but the majority case is that we don't build either and we sit around and argue and bicker and we go slowly. What are we going to do to naturally, organically every day orient a larger group of people in the right direction and see if our tactics are generating progress over time for a customer from their perspective? And metrics on the left and a series of tweets, uh, on the right is like a pretty great com- combo. So here's an example. A couple years ago I, I'd been working in our, our payments group at Stripe for a bit and then I, I started working on, uh, some of our banking and, and incorporation services. It was so... In Atlas, when I started working on it...It had had some success. It had already existed for four or five years pri- prior to me spending time on it. But when I started to look at the support tickets, people were pretty unhappy, frequently. They had a DocuSign stuck in their email box. They needed a co-founder's address, but they didn't know their co-founder's address. They couldn't log into the dashboard to figure out their 83 (b) manual filing instructions. And we saw this, you know... Basically, in the first week of spending time on Atlas, I was just like, "Just show me all the support tickets. Uh, are they happy support tickets?" So people writing in being like, "Oh, I love this service. It's absolutely fantastic. Can you just do A, B, and C more for me?" Or they're, like, sad support tickets. I'm like, "Oh my God, they're all sad support tickets." And so we're just kind of asking ourselves, "Well, why would... why would someone recommend Atlas to a friend?" I was like, "Well, it would have to accomplish A, B, and C activities for them." You know, it'd have to get their company, it would have to handle getting their tax ID from the IRS, it'd have to handle all the downstream administrivia. But surely, if they had a bunch of support tickets at the end, they're not gonna go tell their friends to use this thing. We could measure all of the intermediate parts. We could measure the success rate and the frequency of incorporation services, and we, and we do all those things. But if you, like, looked at the support tickets, there's just no way, if you had a support ticket, you would recommend it to a friend. And so we suggested this metric to ourselves. Companies that have no support tickets through the, through the incorporation service, the whole process, from the moment you start the application open, actually the begi- like, the first page load at the very beginning, all the way through the process, waiting for the government, waiting for the IRS, and we give you two more weeks to write into support. We give, like, an extra buffer two weeks. And if you get through that whole thing with no support tickets, that's a yes. If you have any number of support tickets, that's a no. And we just looked at the percentage of founders that were going through the service with zero support tickets, which is very different than looking at an average, right? You could, you could have the average as 0.3, but that doesn't necessarily mean that they're, you know, that getting to 0.2 is going to cause them to tell their friends more. And we looked, and only 15% of founders were getting through Atlas with zero support tickets through that metric. And I just thought, "Okay, well, let's just drive that number way up. And let's look at the support tickets, decide what people are needing, and we'll bake it into the product, and presumably it'll fix it." People like that more and then tell their friends. And over about 18 months, we took that number from 15% to 85. Uh, we basically just flipped it. And you can look at the market share plotted on the same e- on the same time frame, and it's like the same shape.

   3. LRLenny Rachitsky52:03

      Mm-hmm.

   4. JWJeff Weinstein52:03

      Uh, and I think you have to find a measure by which it speaks directly to the, to what the customer wanted, and that if you, by accidentally leaked your dashboard to them, your customer would be ecstatic to learn that that's what you were measuring the whole time. You know, if we were to showcase the internal Atlas metrics, which we often just, uh, screenshot and publish, I think they'd be pretty happy to hear that we were sp- spending all of our time making sure that none of them had support tickets. And it was incredibly encouraging, uh, and motivating to the engineers on the team, because we could just assign them a topic. "Hey, here's... L- look at all these support tickets. Why don't you come up with the product spec, the scope, the solution? Oh, want to learn more? Just reply to the support ticket email, figure out what they need it, what they need." And so we kind of turned all of the engineers on the team into PMs to go and s- o- like, one issue at a time, figure out what needed to change and build products for it. And that's where we pushed forward on A3B elections, o- automating it, sending it in the mail for folks. We built our own signing service. We turned everything into a click. We, we did, just did sort of the o- the obvious things we saw in the tickets. But as the PM, I was able to just sort of, not on autopilot, but really sit back and have contentful conversations with engineers who are bringing solutions and ideas for product, rather than one person going through all of the potential ideas, scoping them, and assigning them. And because it was based in what people were saying and wanted, it was very motivating for everyone on the team. So somewhat long answer, but figuring out something in which every day we can wake up and look at the same metric and, with some confidence, know what to do is so much better than, "Let's figure out what to do each month," and kind of, like, starting from scratch.

10. 54:08 – 58:23

    Empowering teams with a single metric

    1. JWJeff Weinstein54:08

    2. LRLenny Rachitsky54:08

       I think this is amazing and important advice. Just the power of a single metric that everyone on the team can understand, rally around, and use to prioritize the work they're doing. I've seen exactly the same sort of impact. Funny enough, Airbnb, one of the teams actually had a metric sort of like this, basically reducing the people contacting Airbnb with support issues. Uh, but what (laughs) ended up happening is their team started just making it harder to contact support (laughs) because they're like, "Mm, maybe they don't need to contact support about all these trivial issues, so maybe let's not... let's encourage them to figure it out themselves." Is there anything you've learned about try to avoid these kind of second order effects that are kind of perverse incentives of a metric?

    3. JWJeff Weinstein54:47

       We look at multiple metrics, but we will optimize around one, and you have to use your own judgment to, to look at some of these countermeasures and, and pick them. Um, we would also...You know, that would sort of be our overarching metric for a year, but then we would pick specific tactical metrics about how we would accomplish it. So just, uh, an example that both, i- is how we solved a problem, but also it's just, like, a style of metric that was useful to us. Of c- of course some of these support tickets included things like, "I'm waiting to hear back from Atlas about if they're going to approve my application," because Stripe is required for very good reasons to evaluate certain business types or, um, sanctions lists and so it's a worldwide product. So there is some, you know, incred- incredibly, should be incredibly quick, but, uh, there is a bit of a, like, a r- a review process. And of course if you were not hearing back from us, you would be upset. You're trying to make your company immediately. This is ridiculous. Like, let's get back to us quickly. And so we knew that one of the reasons that people were writing to support was like, "Hey, what's up with my review? What's going on?" And we knew that our tactic was just to drive up how quickly we got to a final decision on folks, and to reduce the number of overturned rejections where someone writes back in saying, "Hi," like, "Come on, I'm totally just thinking something reasonable. What's up? Why did you reject me?" And so we would pick these overarching single metric, which was, which was the companies with zero support tickets, but then we would have a specific KR that was owned by an engineer, which is the tactic that we're going to do, and so we would not allow ourselves the perverse tactics to sort of just casually exist. We would choose which tactics we're going to do and then set a metric for it. And the one... The other reason I love this metric is it's a, it, it's a cohort metric by which you're trying to drive something up and to the left. I, I... Sometimes people will get excited about a chart that goes up and to the right. It's kind of a meme. "Oh, that's going up to the right." I'm really excited about charts that go up and to the left. Uh, so you have to figure out some optimization that you're trying to maximize. And so in this particular case of this risk review, we would look at, hey, for the cohort of customers that started last month, how quickly did we get them to their final risk review by a number of days since they submitted? And so you want a chart that looks a lot like... Just, here we go, you know, right up to the top, because you want 100% of your customers to get their final decision as quickly as possible. And once you know it, when we looked at the chart, it was doing this, right? And so each month, we would just make it a little better, a little better, a little up and to the left, up and to the left, up and to the left, and now basically 100-ish percent of people get their risk review within an hour from, you know, a l- a long tail taking a long time. And so we would constrain the tactic through a metric and then kind of, like, watch it through an optimization function, and then when we got to a point where we were happy with the target, we could put down the tactic. That's another, like, really useful thing about metrics is just, like, you can decide when to stop a tactic because you get to some level of success that you're comfortable with and you can always choose to pick it back up later.

11. 58:23 – 1:05:10

    Picking the right metric for your audience

    1. LRLenny Rachitsky58:23

       So there's some really cool lessons here, um, of just how to pick a good metric, just to kind of maybe summarize what I'm hearing, is you kind of work backwards from essentially MPS, like how many... Why aren't people recommending Atlas? You found, okay, well, people that are complaining and having issues with support and running into problems most likely are gonna be detractors, not gonna wanna recommend Atlas, so let's, let's have those really ambitious goals. Uh, eventually nobody has a support ticket/let's just check how many people have zero issues, and then you identify, okay, what's driving a lot of these support tickets? Okay, it looks like this risk timeline to, like, get to this certain milestone. Let's make that our goal for the next quarter or whatever, and let's focus there and then make an impact and then I imagine move on to other levers within this, uh, bucket of zero contact.

    2. JWJeff Weinstein59:08

       You have to pick the right metric for your audience. In... I wouldn't, like, I wouldn't fully export that metric to everyone.

    3. LRLenny Rachitsky59:18

       Yeah.

    4. JWJeff Weinstein59:18

       Though it's not a terrible one-

    5. LRLenny Rachitsky59:20

       I understand.

    6. JWJeff Weinstein59:20

       ... uh, to export. But in-

    7. LRLenny Rachitsky59:22

       (laughs)

    8. JWJeff Weinstein59:22

       ... in the founder, founders choosing where to get started mindset, again, this isn't just deeply spending time listening to your customers, they all ask their friends, "Hey, how did you start your company?" They want to talk to an older sibling of sorts about how it went. And so we decided that our go-to-market strategy would be to delight our current customers such that they would tell their friends. And for other businesses, I mean, that's always somewhat useful, but you can also reach people with billboards and Google Ads and other types of upsells. That's very difficult in the moment that someone is starting a company, that sort of natural geographic can you get the picture of the bird at exactly the right time, and how are you even supposed to go find people who are about to start companies? Thankfully, they have this practice of just asking their friends, and so if their friends loved it, they're gonna just recommend it, and that, that has... That the metric and the tactic and the go-to-market all lined up, uh, rather than sometimes in cases you might hear someone say, "Well, let's make the product quality better." Well, we could all make the product quality better, but why? Like, why is this actually gonna move what our customers want and the business forward? And when you can line them all up, it, it can be quite, quite beautiful, just especially when you can see it month over month for a long period of time. One other metric I think is j- I, that I would export to just anybody is if you are unsure what to measure, we have this... I don't know if we stole it from somebody else or if we came up with it internally, whatever, is just users having a bad day, where we will just omit a log line any time...We think that a user bumped into a problem, so maybe they hit a 404, or maybe their payout was one day after the, the, uh, the ETA, or they had more than 10 payment declines. You can kind of brainstorm, again, or just listen to customers, w- what, what would cause you to personally have a bad day? And then just omit an event when that occurs, and then you could just make a bar chart, a stacked bar chart of all the bad day reasons, and the frequency by which they're happening, and it is eye-opening to see those frequencies. And it's kind of a metric I hadn't thought about until Stripe Scale, in which you just don't know what's happening until you omit the log line and count it. Like, the frequencies could be kind of mind-blowing, and I think for almost any scale business, if you are bored one day and you're not sure what to measure, just make a users having a bad day chart, omit a log line, and count it as a bar chart, and then anybody else can add their own bar chart on top of it. And so it's become a way for teams to scale their understanding of users through metrics by just saying, "Hey, look, anytime anybody has an idea about why a customer's having a bad day, just omit a log line, like, put it on this chart, and then w- we can choose over time which bad day reasons we want to burn down and hopefully just, like, eradicate them, not just, like, minimize them." But it al- it gave, it gave us kind of a background noise counting system for where there are problems. And any time there's an incident or some customer issue, the first thing I think is, "Ooh, like, I wonder if we have a bad day? I wonder if we have a bad day reason for this? And if we do, I actually feel okay." I'm like, "Oh, yeah, it is a bad day for this customer, I wish they didn't have it, but at least we're aware, and we can evaluate it against other bad days that we want to burn down." What, what does sometimes a little bit grind my gears or, or gives as an opportunity is, like, when we didn't know about that bad day, and it's a surprise to us too. That is, like, to, for me is immediate action. It's like, "Okay, cool. We have to figure out a way to count this bad day. We gotta get it on the chart. That way, we can make an informed decision about, you know, when to invest in, in improving it."

    9. LRLenny Rachitsky1:03:35

       I love this idea. I haven't heard this before. So the idea is just make a list of what happens to a potential customer that would cause them to have a bad day.

    10. JWJeff Weinstein1:03:42

        Yeah.

    11. LRLenny Rachitsky1:03:42

        What are, what are some examples for Stripe or Atlas that you guys have-

    12. JWJeff Weinstein1:03:45

        Are you a Stripe customer, Lenny?

    13. LRLenny Rachitsky1:03:47

        I am. I check my... When my newsletters, uh, the payments go through Stripe, I check my dashboard every day.

    14. JWJeff Weinstein1:03:51

        Cool. So what would be a bad day for you?

    15. LRLenny Rachitsky1:03:53

        Oh, I see. I see some silence, uh, (laughs) happening here. Um, you know, it not loading. The numbers taking a long time to show up. Something being completely off in there. Um, not being able to log in. (laughs) The silence is good. (laughs) I just wanna keep going.

    16. JWJeff Weinstein1:04:19

        The question is, how much... You know, to the audience is, how much will, of the silence will be edited out before-

    17. LRLenny Rachitsky1:04:23

        (laughs) Yeah.

    18. JWJeff Weinstein1:04:23

        ... before it, before it goes live? But this is what I'm talking about, right?

    19. LRLenny Rachitsky1:04:26

        Yeah.

    20. JWJeff Weinstein1:04:26

        Is, is, is I could guess them, right?

    21. LRLenny Rachitsky1:04:28

        Yeah.

    22. JWJeff Weinstein1:04:29

        But c- and I, I can... As you were saying those, I know the, like, I know the URL of those charts, right? I, I, I know the login one, 'cause I, I feel that too, 'cause I've played with Stripe all the time, and then you get a 2FA too frequently-

    23. LRLenny Rachitsky1:04:43

        Mm-hmm.

    24. JWJeff Weinstein1:04:44

        ... and, you know, come on, I had the same cookie. I was just here yesterday.

    25. LRLenny Rachitsky1:04:47

        (laughs)

    26. JWJeff Weinstein1:04:47

        Right? So, we count all that stuff and try to make it better over time. But I'm so excited when someone brings me a new bad day I hadn't thought about yet, uh, because that's just, that's like product catnip.

    27. LRLenny Rachitsky1:05:00

        I love it. And your advice here is, this doesn't necessarily have to be your goal or metric, it's just start watching this thing, 'cause that could lead to a lot of interesting-

    28. JWJeff Weinstein1:05:09

        Yeah.

    29. LRLenny Rachitsky1:05:09

        ... um, ways of operating.

12. 1:05:10 – 1:11:33

    The importance of metric hygiene

    1. LRLenny Rachitsky1:05:10

    2. JWJeff Weinstein1:05:10

       Couple other just, like, quick metrics things-

    3. LRLenny Rachitsky1:05:12

       Yeah.

    4. JWJeff Weinstein1:05:12

       ... just 'cause it's a bit of a... I don't know. Some people like cycling or something, I guess I like metrics. I, I-

    5. LRLenny Rachitsky1:05:20

       (laughs)

    6. JWJeff Weinstein1:05:20

       Like, people, people get a really nice bike, like I want-

    7. LRLenny Rachitsky1:05:23

       Or snacks.

    8. JWJeff Weinstein1:05:23

       ... a really great metric. Um, picking metric titles that make you feel something. So, we could have called that measure of companies, you know, number of companies that do not send a support ticket over X period and Y period, you know, with min me- You know, you see, you sometimes see these charts where the metric itself kind- like, named itself. This is just companies with zero support. That's it. And it, the brevity, and the focus, and the customer mindset built into the chart name can become currency inside the company. It's like, "Oh, I'm working on making the, like, this chart go up," and it feels good to just say the name out loud rather than some com- complicated underscores and mins and maxes and, you know, the database field name is, like, still in the chart title. Uh, these, these are aesthetic choices, but I think make dramatic differences in the cultural willingness for people to buy in and get excited about it, and reduce the need of a product person to just remind everyone every day why they're doing it. It's like the metric is motivating us because it's, it's a motivating thing to talk about. Um, and, and then lastly on metrics is there's just good hygiene that people should bring to their measures. Percentages shouldn't have 41 bl- th- you know, significant digits if, if only two of the digits are relevant. Uh, you should keep all the measures of your dashboard on the same X axes. These are just...... stylistic things that increase the frequency that people want to just wake up every day and open the dashboard and look at it, and that is so powerful if your whole team is looking at the same set of information that has the heartbeat of the customer every single day. In fact, we can measure at Stripe the usage of our dashboards by team, and so we can see which teams are themselves looking at their own metrics, uh, and that is an incredibly useful predictor of how on the same page they are and how, and how customer-obsessed they are. So I just think it's... it's, it's not an area that's sort of behind the scenes, bean-counting, reliability only, is the machine hot. You can make metrics that mean something to the customer and you would be proud if they were to be screenshot and put on the internet and be like, "Wow, I feel like that company is taking their promise to me seriously and I can see myself on those metrics as a customer." That's... that's where we're like, we're really shooting for.

    9. LRLenny Rachitsky1:08:18

       So Tamir, back what you just said there, which I think is a subtle point potentially, is just making the dashboard look nice. Like, the hygiene of the naming conventions and the decimal points and the chart. You're... In your experience, you find that really powerful and important.

    10. JWJeff Weinstein1:08:34

        Couple years ago, Stripe did internal work to make an internal m- metrics kind of dashboard system, and we have a special place called Go Metrics. Y- Many people use the Go slash service where you can just quickly go to a URL. And, again, I, I have to sometimes do a rule rather than a policy. My rule is if it's not on Go Metrics, I'm not going to look at it. So if pe- people can send me one-off queries or charts or screenshots or presentations or emails of charts and data, but that is in the wind. You know, we can't interrogate the query. Metrics are almost always wrong for many weeks. We didn't quite get the definition correctly. You have to live in the metric for quite a while before you really believe in it. And so if you're always looking at some one-off version, it's just very difficult t- for it to rise up to the level of importance that is a thing we should trust to help us decide what to do. That's an incredibly high bar. And so I, I just find you have to look at it frequently enough, and if you're gonna look at it frequently enough, it means it needs to be in a discoverable place.

    11. LRLenny Rachitsky1:09:50

        Mm-hmm.

    12. JWJeff Weinstein1:09:50

        Um, it c- you almost go through a couple stages of grief about it because I'll... we'll kind of put a metric up in a, in a place and everyone initially is like, "Wow, this is great!" Like, "I was so excited to finally see this." And then a couple days later, "Huh," like, "I don't quite understand it. What does that, like, actually mean? I... I saw this other metric from this other angle that kind of makes it feel, y- you know, counterintuitive that it's like this." And then you kind of look into it, "Wait a second, we've been counting it wrong the whole time. Oh, no!" And then you look at it the third week and it's a completely different version, and then you hope that we forget about it (laughs) and maybe we'll go on to something else. They're like, "Nope, we're not gonna forget about it." Week four comes around, you're like, "Wait a second. Okay, this is like..." Then the team meeting starts. "Hey, just a reminder, let's just bring up the metric again, not a screenshot of it, go to the URL." And just that level of frequency and specificity and ritual around it is what brings it into the decision-making c- culture. And again, we treat it the exact same as we do tweets. It's... it's... it's the quantitative and qualitative right next to each other. And because you're putting that amount of attention to it, you can't have a thousand metrics. We just don't physically have time in the day to bring that level of care and understanding to so many things. So then it forces you to winnow so many things you could care about down to a small number of things that you really must care about. And again, that practice goes back to do you really understand what your customer wants? And so for all these tactics, it's about finding out what your customer wants and then different versions of, of how do we sort of know it's true over time and are our tactics improving it?

13. 1:11:33 – 1:37:20

    How Stripe uses “study groups” for product improvement

    1. JWJeff Weinstein1:11:33

    2. LRLenny Rachitsky1:11:33

       Sounds so simple when you describe it that way.

    3. NANarrator1:11:36

       This episode is brought to you by Anvil. Their document SDK helps product teams build and launch software for documents fast. Companies like Carta and Vouch Insurance use Anvil to accelerate the development of their document workflows. Getting to market fast is a top priority for product teams, and the last thing that you or your developers want is to build document workflows from scratch. It's time-consuming, expensive, and distracts from core work. You could stitch together multiple tools and manage those integrations, or you can use an all-in-one document SDK. Most product managers will tell you paperwork sucks. Anvil's document SDK helps teams get to market fast, incorporate your brand style, and give you back time to focus on your company's core differentiated features. For your users, paperwork often starts with an AI-powered web form styled and embedded in your application. From there, you can route data to your backend systems and to the correct fields in your PDFs via API. Complete the process with a white labeled eSignature. The best part about Anvil is the level of customization their SDK provides. Non-technical folks love Anvil's drag-and-drop builder, and developers love their flexible APIs and easy to understand documentation. Build document software fast with Anvil. That's useanvil.com/lenny to learn more or start a free trial. That's use A-N-V-I-L dot com slash Lenny.

    4. LRLenny Rachitsky1:13:05

       Kind of along these same lines, there's something that, uh, I've learned you started at Stripe to help people obsess with user experience and get quality to where it needs to be and move a lot of these metrics, something called study groups.Talk about what that is. I'm very curious.

    5. JWJeff Weinstein1:13:19

       Okay. Let's imagine we've, we understand our customer, we've understand their burning problem, we've built a solution that's in their hands they're using. It's hopefully better than their alternatives and they're start- they're starting to use it, and we have some real traction. You could still mea- measure the success, you could still look at the tweets, and then, of course, you go to pick it up yourself and you're like, "Wait a second. This thing is horrible. (laughs) You know, how, how did it get so bad?" A-a-anyone who's built a product that has gone over the horizon of it's actually in production and being used for some, you know, a year or more, a-as you go to iterate on it, especially when you're in a larger company and there's m-multiple teams and multiple pr- products going on, there's just some entropy in the world that causes these things to go bad. Uh, I actually have a hard time naturally explaining it. You kind of think to yourself, "Well, it's code, it just must be running the same every day." But somehow you do (laughs) enough things in a row and what was once a smooth end-to-end flow for accomplishing a task for our customer is all of a sudden some Byzantine, complicated, broken mall that where all the doors are busted and you have to know exactly the way to get through the dashboard to s- to solve your problem. You're like, "Wait a second, we just, I, I thought this was great just a second ago." So all, I, I mean, many of us have experienced that. Now, okay, what, wh- what are we going to do about it? Well, one is, it is really difficult to take time during the day to allow yourself to even know that this is true, because if you're working on one particular team, you're going to have some next thing you're shipping, you're going to have your customers, you're going to be doing great product work. Meanwhile, your current thing kinda starts to rot in the world, and, uh, decrease all the trust you've earned t- to build what you're building now. It is actually difficult to decide, well, what hours during the week am I going to blo- like block off from my future progress to, to, to, uh, to see it from the customer's perspective today? And there are various techniques to try to incentivize or to structure m- a group of people to do that. Stripe has a thing called friction logs as well, which is an, a, a single individual will pretend to be a customer and go through a product experience end-to-end and write it down. And that has been quite successful at Stripe for very motivated people who can block off the time and have the wide enough context to g- to, to do a complicated thing end-to-end, and have the time to write it up, and the, sort of position the company to send it out as a critique to, potentially to not just your own team but, but in, in a whole organization. So that's actually like a pretty high bar to cross over. So I was kinda brainstorming, what could we do to make this more fun and have the frequency by which we're looking at our own products dramatically increase? And we kind of iterated through a throu- through a, few a through ideas, and I landed on this thing called study group, which is basically a random group of people inside the company. They might not work on any particular team, they might not all be PMs. Just, like, literally anyone who wants to sign up, a support person, a salesperson, someone who's, you know, on our events team, an engineer in our infrastructure stack, anybody can sign up for a study group. We show up, maybe four to eight people total. We pretend to be some company with some outcome problem. Maybe we want to accept money in person at an upcoming, uh, farmer's market, or we want to run a multi-country global business where we have software that another business would use to run their business. Like, it would be something quite, quite complicated. We could pick any motivating goal. And there are two rules to study group. Rule one is you do no longer work at Stripe. You, you're no-, you're not, you do not work at Stripe. Not pretend you work, don't work at Stripe, not try to forget, you do not work at Stripe. You've never worked at Stripe. You work at... And we make up a name of the company, you know, Dolphin Aquarium Industries, and we will pick a CEO of the company from the, from the group in, amongst us, and okay, J- Jenny is the CEO. "Hey, Jenny," like, "What, what do you wanna do today?" Like, "Well, I wanna sell in-person tickets to the Dolphin Aquarium." It's like, "Cool. Where would you start?" Right? So we actually embody the customer. We will not break character. It's a little bit of the improv thing. And as the kind of maestro of study group, I will firmly but kindly, if I hear you use any internal Stripe knowledge, which is natural to do, you're like, well, 'cause you work on a team and you, you know where that button really is and you know the docs link goes a little bit to the wrong place, but if you click the other link, you'll get to it. If I can see you use any internal Stripe knowledge, I'll just pause and say, "Hey, let's re-, let's redo the sentence or redo what we did with, with no Stripe knowledge. As a reminder you don't work at Stripe." And it takes us a couple times, but people really get into it. And all of a sudden, uh, we'll start making up characters and, you know, the CEO will be like, "Oh, uh, Tim, uh, you're our designer. You know, where are we gonna get the color palette?" All of a sudden (laughs) there's a person who's not a designer, right, in real life, is now all of a sudden having to practice the empathy-

    6. LRLenny Rachitsky1:19:19

       Mm.

    7. JWJeff Weinstein1:19:20

       ... from, to be in the customer's position. And because we're gonna be doing it for an hour, hour and a half at a time-You start, you actually start to believe that you don't work at Stripe and you, you work at do- Dolphin Aquarium Industries, and you start to really feel it. So that's rule one, you don't work at Stripe. And rule two is, we're not here to solve any problems. We're not here to critique. We're not here to solution or suggest. We're not here to file bugs. All of those things is recorded, we can do that later, whatever. This is just about practicing empathy for the customer and going through the product. And we have done more than 25 of these at Stripe thus far in the last, just this year, last few months, of, of 2024. Uh, more than 250 people, uh, have gone, li- like participate in the study group, and it is deeply eye-opening for those involved. The responses are, you know, sort of business emotional, you know? (laughs) Not, not, not, like, not super emotional, but just, wow, I, I, I, I sort of can't believe how hard it was to accomplish X, Y, and Z. I thought we were amazing at blah. It's like, well, we actually are still pretty good, but we need to get back towards amazing. Or, wow, I didn't even know that s- internally people would know that our products did one, two, and three. And this has become so internally popular that teams have adopted it for themselves. And so we've kind of franchised Study Group internally already, where different teams will, will, will run it. Um, but, but I think we're gonna continue it, because it, I think a little bit is coming out of the pandemic. People just want more group activities. I think some of it is the slowness by which we do it. We're not rushing through it, um, which is another reason I appreciate your podcast as well. It's like, let's, like, really get into the details and not be rushed for time. You, you forget that you have a meeting at the end ki- kind of amount of time. No one is to blame or defensive, because it's not your product at all. It's a random group of people. We don't even introduce ourselves, about what team we worked on. Uh, we're just all here as participants to embody the customer. And then I think lastly, and, and a reason why it's, it's, it's worked, and these are, many of these have just been actually super surprising to me, I wouldn't have been able to predict ahead of time, is, it is just fun. It's just fun. People want to make up the name of the company. They're excited about what Dolphin Industries would sell. Uh, a person who's not a CTO in real life gets to pretend to be a CTO. And it, it just, it, uh, there's a little theatrics to it, but it, it has been different enough from our, our, our, our other approaches to, for a company like Stripe, which is already, like, pretty focused on this topic, that we think that it's, there's, there's enough legs here that we're gonna keep it, and, and kind of invest in groups pretending to be the customer, and going through it painstakingly slowly, rather than only demos or only writing, which has other benefits, but misses the participation of a larger group and, and more fresh eyes. So it's been, it's been fun, and we named it Study Group, I guess I named it Study Group, because I wasn't sure what it would be initially. So, I just came up with something that sounded cutesy, uh, but, uh, gave me enough leeway that we could kind of adjust it over time, and this is where we've landed.

Episode duration: 2:34:59