Things That Don't Scale, The Software Edition – Dalton Caldwell and Michael Seibel

1. 0:00 – 0:56

   Do things that don't scale

   1. MSMichael Seibel0:00

      We'll get a founder that's like, "Oh, how do I, like, test my product before I launch to make sure it's gonna work?" And, and I always come back and tell the founder the same thing. Like, if you have a house, and it's got full of pipes, and you know that some of the pipes are broken and they're gonna leak, you can spend a lot of time trying to check every pipe, guess whether it's broken or not and repair it, or you can turn the water on.

   2. DCDalton Caldwell0:23

      [laughs]

   3. MSMichael Seibel0:24

      And like, you'll know. [laughs] Like, you'll know exactly the work to be done. Hey, this is Michael Seibel with Dalton Caldwell. Today, we're gonna talk about what does it mean to do things that don't scale, the software edition. In this episode, we're gonna go through a number of great software and product hacks that software companies used to figure out how to make their product work when perhaps they didn't have time to really build the right thing. Now, Dalton, probably the master of this, um, is a person we work

2. 0:56 – 1:35

   90/10 solution - Paul Buchheit

   1. MSMichael Seibel0:56

      with, a guy named Paul Buchheit, who invented this term, the 90/10 solution.

   2. DCDalton Caldwell1:00

      He always says something like, "How can you get 90% of the benefit for 10% of the work?" Always. This is what he always puts onto people when they tell him it's really hard to build something or it'll take too long to code it. Um, he'll just always push on this point. And-

   3. MSMichael Seibel1:15

      Yep

   4. DCDalton Caldwell1:16

      ... you know, founders don't love it. [laughs] Right? Would you, would you say that's a, a fair assessment, Michael? Um-

   5. MSMichael Seibel1:22

      That's a fair assessment, yes. Founders hate it.

   6. DCDalton Caldwell1:25

      They ... But, but tell, tell the, tell the audience why it's worth listening to the guy. Like, why does he have the credibility to, to say that to people?

   7. MSMichael Seibel1:32

      Well, PB is the inventor

3. 1:35 – 5:40

   Gmail

   1. MSMichael Seibel1:35

      of Gmail. And as kind of a side project at Google, he invented something that 1.5 billion people on Earth actively use, and he literally did it [laughs] doing things that don't scale. So-

   2. DCDalton Caldwell1:50

      Yes

   3. MSMichael Seibel1:51

      ... I'll start the story, and then please take it over. So as I remember it, um, PB was pissed about the Gmail product ... uh, Google product he was ... Sorry. The email product he was using. And so, um, Google had this newsletter, um, product. The first version of Gmail, he basically figured out how to put his email into this Google Groups UI. And, um, as he tells the story, kind of his eureka moment was when he could start reading his own email in this UI. And then from that point on, he stopped using his old email client. And what I loved about this is that as he tells the story, every email feature that any human would wanna use, he just started building from that point. And so, you know, he would talk to YC batch, and he's like, "And then I wanted to write an email."

   4. DCDalton Caldwell2:46

      [laughs]

   5. MSMichael Seibel2:46

      "And so I built [laughs] writing emails." And if you know PB, like, he could have gone a couple days reading emails without replying at all, so [laughs] like-

   6. DCDalton Caldwell2:54

      That's right

   7. MSMichael Seibel2:54

      ... he didn't need writing emails to start. I remember him telling the first time he got his, like, coworker, like literally, like, his desk mate or something, to try to use it. And h- his desk mate's like, "This thing's pretty good. It loads really fast. It's really great. The only problem is, PB, it has your email in it-"

   8. DCDalton Caldwell3:12

      [laughs]

   9. MSMichael Seibel3:12

      "... and I want it to have my email." [laughs] And PB was like, "Oh, shit. Okay."

   10. DCDalton Caldwell3:16

       That's a bug.

   11. MSMichael Seibel3:16

       "Well, I gotta build that. I'm gonna build that." [laughs]

   12. DCDalton Caldwell3:20

       I forgot about that.

   13. MSMichael Seibel3:20

       Perfect 90/10 solution. [laughs] And so then it started spreading throughout Google. And do you remember when it broke?

   14. DCDalton Caldwell3:29

       No. What, what happened?

   15. MSMichael Seibel3:30

       Oh, so he told this story where, like, one day PB came in late to work, which is, you know, knowing PB-

   16. DCDalton Caldwell3:36

       Every day.

   17. MSMichael Seibel3:37

       You know.

   18. DCDalton Caldwell3:37

       [laughs]

   19. MSMichael Seibel3:38

       And everyone was looking at him, like, really weird, and they were all, like, a little pissed. And he got to his desk, and someone came over to him and was like, "Don't you realize that Gmail's been down, like, all morning?" And PB was like, "No, I just got to work. I didn't know." [laughs]

   20. DCDalton Caldwell3:53

       [laughs]

   21. MSMichael Seibel3:53

       And so he's, like, trying to fix it, trying to fix it, and then his coworkers see him, like, grab a screwdriver and go to the server room. And it was like, they were like, "Oh, God. Why did we trust PB with our email? [laughs] Like, we're totally screwed." And I think he figured out, like, there was a corrupted hard drive. And I remember that point of the story, he was like, he says, "And that day I learned that people really think email's important, and it's gotta always work." [laughs] It's like-

   22. DCDalton Caldwell4:22

       And like, what I like about these-

   23. MSMichael Seibel4:23

       ... perfect 90/10 solution

   24. DCDalton Caldwell4:24

       ... is I think the reason, I think the reason he did it, man, is because he liked to run Linux on the desktop-

   25. MSMichael Seibel4:31

       Mm

   26. DCDalton Caldwell4:31

       ... and he didn't wanna run Outlook. Like, the, the Google, like, suits were trying to get him to run Outlook on Windows, and he was like, "Eh, I don't really wanna run Windows." But yeah, it was the dirtiest hack. And as I recall, in this, you know, final part of the story, it was hard for him to get Google to release it because they were afraid it was gonna take up too much hardware. And so there was all of these, there was all these issues where there's a good ... There was a decent chance I think it never would've been released.

   27. MSMichael Seibel4:55

       Well, this part was that everyone thought Gmail's, like, um, invite system was, like, some cool, like, growth hack.

   28. DCDalton Caldwell5:03

       Virality hack, yeah.

   29. MSMichael Seibel5:03

       Like, virality hack. It's like, oh, if you got access to Gmail, you got, I think, four invites to give someone else, and these are, like, precious commodities. And it was r- it was another product. It was just another version of things that don't scale. They didn't have enough, um-

   30. DCDalton Caldwell5:17

       Hardware
4. 5:40 – 10:05

   Facebook

   1. MSMichael Seibel5:40

      another story about Facebook early days that, that is-

   2. DCDalton Caldwell5:42

      Yeah

   3. MSMichael Seibel5:42

      ... similar in this light.

   4. DCDalton Caldwell5:44

      So, so let me paint the picture. Um, back when you started a startup a long time ago, you had to buy servers and put them in a data center, which is a special room [laughs]

   5. MSMichael Seibel5:56

      [laughs]

   6. DCDalton Caldwell5:56

      ... that's air-conditioned that just has other servers in it, and you plug them in, and they're, it's, they have fast internet access. And so being a startup founder until AWS took off- Part of the job was to drive to the suburbs or whatever, drive to some data center, which is an anonymous warehouse building somewhere-

   7. MSMichael Seibel6:17

      Mm

   8. DCDalton Caldwell6:17

      ... go in there and, like, plug things in. And what was funny is when your site crashed, it wasn't just depressing that your site crashed, it actually entailed getting in your car.

   9. MSMichael Seibel6:28

      Yeah. [laughs]

   10. DCDalton Caldwell6:29

       [laughs] Like, part of being a startup founder was waking up at 2:00 AM and getting in your car and driving to, like, Santa Clara-

   11. MSMichael Seibel6:37

       Yep

   12. DCDalton Caldwell6:38

       ... because your code wedged the c- you had to physically reboot the server, and your site was down until you physically rebooted the server. So anyway, I'm just trying to set the stage for people. So this was-

   13. MSMichael Seibel6:46

       Yes

   14. DCDalton Caldwell6:46

       ... this was what our life was like, okay? And so my company, Imeem, we had a data center in Santa Clara, [clicks tongue] and there was a bunch of other startups there as well. And so something that I liked to do was to look at who my neighbors were, so to speak. There was never people there, it was just their servers, and there'd be a label at the top of the rack, and you could see their servers and you could see the lights blinking on the switch. Okay? S- so this is what it was like. And so, uh, our company was in the data center, in this data center in Santa Clara, [clicks tongue] and then one day there's a new tenant and, "Oh, a new neighbor." So I look at it, and the label at the top of the cage next to ours, you know, three feet away, the label said TheFacebook.com. [clicks tongue] And I remember being like, "Oh yeah, I've heard of this. Like, cool. Like, sounds good." And they had these super janky servers, I think there was maybe eight of them, um, when they first moved in. [clicks tongue] And they were, like, super cheap, they're, like, super micro servers. Um, you know, like, the wires were hanging out. Like, it di- you know, I'm like, "Cool." But the, but the lights were blinking really fast, okay?

   15. MSMichael Seibel7:48

       [laughs]

   16. DCDalton Caldwell7:49

       [clicks tongue] And so what I remember was that there was labels on every server, and the labels were the name of a university. And so at the time one of them, one of the servers was named Stanford, one of them was named Harvard, you know, like... And it made sense 'cause I was familiar with the Facebook product at the time, which was, like, a college social network that was at, like, eight colleges. Okay? So then I watched every time we would go back to the data center, they would have more servers in the rack with more colleges. And it became increasingly obvious to me that the way they scaled Facebook was to have a completely separate PHP instance running for every school that they copy and pasted the code, the code to. They would have a separate MySQL server for every school, and they would have, like, a Memcache instance for every school. And so you would see, like, the University of Oklahoma, you know, my s- you'd see the three servers next to each other. And the way that they managed to scale Facebook was to just keep buying these crappy servers.

   17. MSMichael Seibel8:46

       [laughs]

   18. DCDalton Caldwell8:47

       They would launch each school, and it would only talk to a single school database, and they never had to worry about scaling a database across all the schools at once. 'Cause again, at the time, hardware was bad, okay, MySQL was bad, like, the technology was not great. If they had to scale a single database, a single users table to hundreds of millions of people, it would've been impossible. And so their hack was the 90/10 solution like PB used for Gmail, which is like, just don't do it. And so at the time, if you were, like, a Harvard student and you wanted to log in, you would... It was hard-coded to the URL was harvard.thefacebook.com, right, man?

   19. MSMichael Seibel9:21

       Mm-hmm.

   20. DCDalton Caldwell9:21

       Like, and so if you try to go to stanford.thefacebook.com, it'd be like, you know, error. [laughs]

   21. MSMichael Seibel9:27

       Mm.

   22. DCDalton Caldwell9:27

       Like, that was just a separate database. And so then they wrote code so you could bounce between schools, and it actually took them years to build a global users table, as I recall, um, and avoid this, this hack. And so anyway, the thing they did that didn't s- scale is to copy and paste their code a lot and have completely separate database instances that didn't talk to each other. And I'm sure people that work at Facebook today, I bet a lot of people don't even know this story. [laughs] But, like, that's what it took. That's the real story behind how you start something big like that versus what it looks like today.

   23. MSMichael Seibel10:01

       [clicks tongue] So in the case of Twitch,

5. 10:05 – 12:07

   Twitch part 1

   1. MSMichael Seibel10:06

      [keyboard clacking] all, if not all, like, most of the examples of this came from this core problem, and it's why I tell people to not create a live video site. A normal website, even a video site, on a normal day will basically have peaks and troughs of star- of, of traffic, and, and the largest peaks will be two to four X the steady state traffic. So you can engineer your whole product such that if we can support two to four X the steady state traffic and our site doesn't go down, we're good. On a live video product, our peaks were 20X. Now, you can't even really test 20X peaks.

   2. DCDalton Caldwell10:45

      Mm-hmm.

   3. MSMichael Seibel10:45

      You just experience them and fix what happens [laughs] when 20X more people than normally show up on your website 'cause some pop star is streaming something. And so two things kinda happened that, that were really fun about this. So the first hack we had was, um, if suddenly some famous person was streaming, on their channel there'd be a bunch of dynamic things that could load, like your username would load up on the page or their channel, and the view count would load up, and a whole bunch of other things that would basically hit our application servers and destroy them if 100,000 people were trying to request the page at the same time. So we actually had a button that could make any page on Justin.tv a static page. All those features would stop working. Your name wouldn't appear, the view count wouldn't update. Like, literally a static page that loaded our video player, and you couldn't touch us. We could just cache that static page, and as many people as possible wanna look at it. Now, to them, certain things might not work right [laughs] but they were watching the video, the chat worked, 'cause that was a different system, the video worked, that was a different system, and we didn't have to figure out the harder problems until later. Later, actually, Kyle and Emmett worked together to figure out how to cache parts of the page while make other parts of the page dynamic, but that happened way, way later.

   4. DCDalton Caldwell12:03

      Dude, that reminds me-

   5. MSMichael Seibel12:03

      The second-

   6. DCDalton Caldwell12:04

      Let me give you a quick anecdote.

   7. MSMichael Seibel12:05

      Yes.

   8. DCDalton Caldwell12:06

      Remember Friendster

6. 12:07 – 15:03

   Friendster vs MySpace

   1. DCDalton Caldwell12:07

      before MySpace?

   2. MSMichael Seibel12:08

      Yeah, of course.

   3. DCDalton Caldwell12:09

      Every time you would log in, it would calculate how many people were two degrees of separation from you, and it would s- it would fire off a MySQL thread Where you would log in, it would look at your friends, and it would calculate your friends of friends and show you a live number of how big your extended network was.

   4. MSMichael Seibel12:23

      Wow.

   5. DCDalton Caldwell12:23

      And the founders, you know, John Abrams, he thought this was, like, a really important feature. I remember talking to him about it. Guess what MySpace's, uh, do things that don't scale solution was?

   6. MSMichael Seibel12:32

      They made it up or whatever.

   7. DCDalton Caldwell12:33

      If they were in your friends list, it would say, "This is in your friends..." You know, "So-and-so is in your friends list." And if it wasn't, it would say, "So-and-so is in your extended network."

   8. MSMichael Seibel12:40

      There it is. [laughs]

   9. DCDalton Caldwell12:41

      That was it. That was the feature. [laughs]

   10. MSMichael Seibel12:43

       [laughs]

   11. DCDalton Caldwell12:43

       And so, so Friendster was, like, trying to, like, hire engineers and scale MySQL, and they're running into, like, too many threads on Linux issues, and, like, updating the kernels. And MySpace was like, "Ah, so-and-so is in your extended network. That's our solution." Anyway-

   12. MSMichael Seibel12:57

       Well, that-

   13. DCDalton Caldwell12:57

       ... carry on. But that's same deal.

   14. MSMichael Seibel13:00

       So our second one was, um... It would always happen with popular streamers. So our second was if you imagine, um, if someone is really popular and there's 100,000 people who wanna watch their stream, we actually need multiple video servers to serve all of those viewers. So we basically propagate the original stream coming from the person streaming across multiple video servers until there was on enough video servers to serve all the people who are viewing. The challenge is, is that we never had a good way of figuring out how many video servers we should propagate the stream to. And if a stream would slowly grow in traffic over time, we had a little algorithm that could work, and, like, spin up more video servers and it'd be fine. But what actually happened was that a major celebrity would announce they were going on, and all their fans would descend on that page. And so the second they started streaming, 100,000 people would be requesting the live stream, bam, video server dies. And so we were trying to figure out solution, solution, solutions and, like, how do we, how do we model this? How do we... Like, there were all kinds of, like, overly complicated solutions we came up with. And then, once again, Colin and Emmett got together and they said, "Well, the video system doesn't know how many people are sitting on the website before the video stream, before it starts, starts trying to serve video." But the website does. All the website has to do is communicate that information to the video system, and then it could pre-populate the stream to as many video servers as it would need to, and then turn the stream onto users. So what happened now in this setup is that some celebrity would start streaming. They would think they were live. No one was seeing their stream while we were propagating their stream to all the video servers that are needed, and then suddenly the stream would appear for everyone, and would look like it worked well. And, like, the delay was a couple seconds. It wasn't that bad, right? But, like, dirty, super dirty, [laughs] but it worked. And, and honestly, that's gonna be kind of the theme of this whole setup, right? Super dirty, but it worked. You had a couple of these in Imeem,

7. 15:03 – 16:59

   Imeem

   1. MSMichael Seibel15:03

      right?

   2. DCDalton Caldwell15:04

      Yeah. There, there was a couple that we had at Imeem. So one of them, um... So at the time, again, like, to, to set the stage, the innovation of showing video in a browser without launching RealPlayer, [laughs] no one here probably knows what that is.

   3. MSMichael Seibel15:20

      [laughs]

   4. DCDalton Caldwell15:21

      But it used to be to launch a video, it would launch another application in the browser that sucked, and it would, like, crash your browser, and you hated your life, okay? So one of the cool innovations that YouTube, the startup YouTube, had before it was acquired by Google, was to play video in Flash in the browser that required a, no external dependencies. It would just play right in the browser. At the time, that was, like, awesome. Like, it was like, it was a major product innovation to do that.

   5. MSMichael Seibel15:46

      Yes.

   6. DCDalton Caldwell15:46

      Um, and so we wanted to do that for music at Imeem, and we were looking at the tools available to do it, and we saw all this great tooling to do it for video. And so rather than rolling our own tools that was music specific, we just took all of the open source video stuff and hacked the other video code that we had so that every music file played on Imeem was actually a video file. It was a .FLV back in the day.

   7. MSMichael Seibel16:16

      Yeah.

   8. DCDalton Caldwell16:16

      And it was actually a Flash video player. Um, and the entire... It was basically a v- we were playing video files that had, like, a zero bit in the video field-

   9. MSMichael Seibel16:27

      [laughs]

   10. DCDalton Caldwell16:28

       ... and it was just audio. And we actually were transcoding uploads into video files.

   11. MSMichael Seibel16:33

       [laughs]

   12. DCDalton Caldwell16:34

       You know what I'm saying? Like, the whole, the entire thing was, was... It was a video site with no video. I don't know how else to explain it.

   13. MSMichael Seibel16:40

       [laughs]

   14. DCDalton Caldwell16:40

       Um, and it worked. And, and I do think this is a recurring theme, is a lot of the best product decisions are ones made kind of fast and kind of under duress.

   15. MSMichael Seibel16:49

       Yes.

   16. DCDalton Caldwell16:49

       I don't know what that means. But it's like when it's, like, 8:00 PM in the office and the site's down, you tend to come up with good decisions on this stuff.

   17. MSMichael Seibel16:58

       So

8. 16:59 – 20:17

   Twitch part 2

   1. MSMichael Seibel16:59

      we had two more at Twitch that were really funny. Um, the first one, talking about duress, um, was our free peering hack. So streaming live video is really expensive. Back then, it was really expensive, and we were very bad fundraisers. That was mostly my fault. And so we were always in the situation we didn't have enough money to stream as much video, and we had this global audience of people who wanted to watch content. And so we actually hired one of the network, um, ops guys from YouTube who had figured out how to kinda scale a lot of YouTube's early, um, uh, usage, and he taught us that you could have free peering relationships-

   2. DCDalton Caldwell17:43

      Yep

   3. MSMichael Seibel17:44

      ... with different ISPs around the world, and so that you wouldn't have to pay a middleman to, say, serve video to folks in Sweden. You can connect yourself, your servers. You, you go... I forgot what the term called was.

   4. DCDalton Caldwell17:54

      Yeah. It saves you money, and it saves them money. That's what they wanted.

   5. MSMichael Seibel17:57

      Yeah. And there were these massive, like, switches where you could basically, like, run some wires to the switch, and bam, you could connect to the Swedish ISP. Now, the problem is, is that some ISPs wanted to do this free peering relationship, where basically you can send them traffic for, for free, they can send you traffic for free. Others didn't. They, they didn't wanna do that, or, like, they weren't kind of with it. And so I, I think it was Sweden, but I don't remember. Some ISP was basically s- not allowing us to do free peering, and we were spending so much money sending video to this country, and we were generating no revenue from it. It's like we, we couldn't make a dollar on advertising. And so what we did is that after 10 minutes of people watching free, free live video, we just put up a big thing that blocked the video that said, "Your ISP is not doing a free peering relationship with us, so we can no longer serve you video. If you'd like to call to complain, here's a phone number and email address." [laughs] And that worked, and like-

   6. DCDalton Caldwell18:57

      How fast did it take for that to work?

   7. MSMichael Seibel18:58

      [laughs] I don't remember how fast. I just remember it worked, and I remember thinking to myself, "It's almost unbelie-" Like, that ISP was a real company. Like, we were like a website [laughs] in San Francisco.

   8. DCDalton Caldwell19:10

      [laughs]

   9. MSMichael Seibel19:11

      Um, and, and hey, that worked. And then the second one was translation. So we had this global audience, and w- we would, like, call these translation companies, and we'd ask them like, "How much would it cost to translate our site into these, like, 40 different languages?" And they were like, "Infinite money." And we're like, "We don't have infinite money." And so I think we stole the solution from Reddit. Um, we were like, "What happens if we just build a little website where our community translates everything?" And so basically it would just, like, serve up every string in English, and it was like served to anyone who came to the site who wasn't from an English-speaking country. And it was like, "Do you wanna volunteer to translate this string into your local language?" And of course, you know, people were like, "Well, what if they do a bad job translating?" It's like, well, the alternative is it's not in their language at all. [laughs]

   10. DCDalton Caldwell19:58

       [laughs]

   11. MSMichael Seibel19:58

       So like, let's not make the perfect enemy of the good. And I think we had something where, like, we would get three different people to translate it and it, like, match, but, like, that happened later. But we basically got translation for a whole product for free. Um, maybe to end, 'cause I think this might be the, like, maybe the funniest of them all,

9. 20:17 – 23:39

   Google

   1. MSMichael Seibel20:17

      tell the Google story 'cause I, I think this one's, like, the, like-

   2. DCDalton Caldwell20:19

      Yeah

   3. MSMichael Seibel20:19

      ... really, like [laughs]

   4. DCDalton Caldwell20:21

      So, so look, for the Facebook story, that was firsthand where I personally witnessed the service with my own eyes, so I'm 100% confident that is what happened because I was-

   5. MSMichael Seibel20:30

      [laughs]

   6. DCDalton Caldwell20:30

      ... it was me, right? This is not-

   7. MSMichael Seibel20:31

      Yes.

   8. DCDalton Caldwell20:31

      This Google story is secondhand, and so I may get some of the details wrong. I apologize in advance. But I'll tell you, this was relayed to me by someone that was there, all right? You ready? So look, the original Google algorithm was based on a paper that they wrote, which you can go read, um, PageRank. Um, it worked really well. It was a different way to do a search, okay? It worked... They always didn't have enough hardware to scale it 'cause remember, there was no cloud back then. You had to run your own servers. And so as the internet grew, it was harder and harder to scale Google. You still with me? Like, there were just more web pages on the internet. So it worked great when the web was small, but then they kept having more web pages really fast, and so Google had to run as fast as they could to just stay in the same place. Just to run a crawl and re-index the web was, like, a lot of work. And so the way it worked at the time is they weren't re-indexing the web in real time constantly. They had to do it in one big batch process w- back in the day, okay? And so there was some critical point, this was probably in the 2001 era. Again, this is secondhand. I don't know exactly when it was. But there was some critical point where this big batch process to index the web started failing, and it would, it would take three weeks to run the batch process. It was like the, you know, reindexweb.sh, you know?

   9. MSMichael Seibel21:55

      [laughs]

   10. DCDalton Caldwell21:55

       It was, like, one script that was like [laughs] do Google-

   11. MSMichael Seibel21:57

       [laughs]

   12. DCDalton Caldwell21:58

       ... you know. And it started failing, and so they tried to fix the bug, and they restarted it. And then it failed again. And so the story that I heard is that there was some point where for maybe three months, maybe four months, I don't remember the exact details, there was no new index of Google. [laughs]

   13. MSMichael Seibel22:15

       Yep.

   14. DCDalton Caldwell22:15

       They had stale results. So anyone, any user of Google, they didn't know that. You know, the users didn't know this. But any user of Google was seeing stale results, and no new websites were in the index for quite some time, okay?

   15. MSMichael Seibel22:26

       Yes. Yes.

   16. DCDalton Caldwell22:26

       And so obviously they were freaking out inside of Google. Um, [laughs] and this was the genesis for them to create MapReduce, which they wrote a paper about, which was a way to parallelize and break into pieces all the little bits of crawling and re-indexing the web. And, um, you know, Hadoop was created off of MapReduce. There's a bunch of different software used, and I would argue every big internet company now uses the descendants of this particular piece of software and that it was created under duress when Google secretly was completely broken for an extended period of time because the web grew too fast.

   17. MSMichael Seibel23:04

       But I think this is the f- most fun part about this story. When the index started getting scaled, stale, did Google shut down the search engine? Did you-

   18. DCDalton Caldwell23:14

       Nope.

   19. MSMichael Seibel23:14

       Like, that's the coolest part. Like, people just didn't realize it.

   20. DCDalton Caldwell23:18

       They didn't know.

   21. MSMichael Seibel23:18

       [laughs] They-

   22. DCDalton Caldwell23:19

       And did they build this first? Again, in terms of do things that don't scale, did they build MapReduce before they had any users?

   23. MSMichael Seibel23:24

       No.

   24. DCDalton Caldwell23:25

       No. [laughs]

   25. MSMichael Seibel23:26

       No.

   26. DCDalton Caldwell23:26

       Like, they basically made it this far by just building a monolithic product, and they only dealt with this issue when they had to.

   27. MSMichael Seibel23:36

       You know, I think this is, like, such a common thing that comes

10. 23:39 – 25:55

    Startup advice

    1. MSMichael Seibel23:39

       up when we give startup advice, you know? We'll get a founder that's like, "Oh, how do I, like, test my product before I launch to make sure it's gonna work?" And, and I always come back and tell the founders the same thing. Like, if you have a house and it's got full of pipes, and you know that some of the pipes are broken and they're gonna leak, you can spend a lot of time trying to check every pipe, guess whether it's broken or not, and repair it, or you can turn the water on.

    2. DCDalton Caldwell24:05

       [laughs]

    3. MSMichael Seibel24:06

       And, like, you'll know. [laughs] Like, you'll know exactly the work to be done when you turn the water on. I think people are always surprised that that's basically all startups do is just turn the water on, fix what's broken, rinse and repeat. And, like, that's how big companies get built. It's never taught that way, though, right? It's always taught-

    4. DCDalton Caldwell24:25

       Yeah

    5. MSMichael Seibel24:25

       ... in like, oh, somebody had a plan, and they wrote it all down. And it's like never.

    6. DCDalton Caldwell24:29

       Well-

    7. MSMichael Seibel24:30

       Never

    8. DCDalton Caldwell24:30

       ... and you earn the privilege to work on scalable things by making something people want first. Uh, you know what I think about sometimes with Apple is picture, like, Wozniak hand soldering the original Apple computer and, like, those techniques compared to, like, whoever it is that works at Apple that designed AirPods.

    9. MSMichael Seibel24:50

       [laughs] Yeah.

    10. DCDalton Caldwell24:52

        Like, it's the same company, but, like, Wozniak hand soldering [laughs] is not scalable.

    11. MSMichael Seibel24:59

        Yep.

    12. DCDalton Caldwell24:59

        But i- you know, they earned, because that worked, they earned the privilege to be able to make AirPods now.

    13. MSMichael Seibel25:04

        Yes.

    14. DCDalton Caldwell25:04

        And because Google Search was so good, they earned the privilege to be able to create super scalable-

    15. MSMichael Seibel25:09

        Yes

    16. DCDalton Caldwell25:09

        ... stuff like m- like MapReduce and all these other awesome internal tools they built, right?

    17. MSMichael Seibel25:13

        Yes.

    18. DCDalton Caldwell25:14

        But if they wouldn't have built that stuff first, it wouldn't be Google, man.

    19. MSMichael Seibel25:18

        And so to wrap up, kind of what I love about Things That Don't Scale is that it works in the real world, right? The Airbnb founder's taking photos. The DoorDash folks doing deliveries. It also works in the software world, right? Like, don't make the perfect the enemy of the good. Just try to figure out any kind of way to give something pe- somebody something that they really want, and then solve all the problems that happen afterwards. And, and you're, you're doing way better. All right. Thanks so much for watching the video. [outro music]

Episode duration: 25:55