Scaling Slack and GitHub with Maude Lemaire

Maude Lemaire 0:00 We needed 500,000 active WebSocket connections. We had a solution that would spin up Slack as a headless Chrome instance in a bunch of Kubernetes pods. It was 37 cents per headless Chrome instance per five-hour tranche of load testing.

Gergely Orosz 0:14 Yeah, we're talking about $150,000 for that one test. [laughs]

Maude Lemaire 0:17 There were 16 shards for all of pull requests and issues.

Gergely Orosz 0:22 Across all of GitHub.

Maude Lemaire 0:23 Across all of GitHub. Only 16, which is not very many given the amount of data that we know is associated with that whole space. The answer was, "Oh, well, we can't have more because we already have so many scatter queries."

Gergely Orosz 0:39 What parts of the job does AI not help you with?

Maude Lemaire 0:42 It has not helped me with performance optimizations, basically at all. At least the investigation part. The LLMs right now I find are good at-

Gergely Orosz 0:56 How do you scale a product that millions of people use daily when the architecture underneath it is starting to break? Today's guest did this both at Slack and GitHub. Maude Lemaire is a principal engineer and the author of the book Refactoring at Scale. She spent eight years at Slack, worked on database scaling at GitHub, and improved time to first token at Cursor. Today, we cover load testing with 500,000 WebSocket connections, a job where using headless Chrome instances were too resource intensive and too expensive, why GitHub had only 16 database shards for all pull requests and issues, and why scatter queries were a massive headache inside a large Rails monolith like GitHub's, why platform and product engineers should rotate between teams, and more. If you want to understand what scaling up looks like inside of products you probably use every day, this episode is for you. This episode is presented by TurboBuffer, vector and full-text search built on object storage. It's fast, cheap, and extremely scalable. This episode is brought to you by Antithesis, the platform to release with confidence. In today's episode, we'll talk about rewriting massive production systems as they scale up, which is the kind of engineering challenge as rebuilding a 747 midair. But why not just use code review when using AI coding agents? It's because code review is an increasingly naive and ineffective approach to do so, which is why Antithesis goes well beyond this practice. Antithesis runs your whole system in a hostile simulation. This simulation includes both targeted testing and fuzz testing. By running this simulation, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it much easier to fix issues. The first thing I thought when I heard about Antithesis is that automated bug discovery and fully deterministic testing sounds like science fiction, but actually, it's hardcore engineering under the hood. Jane Street, Superbase, Rapid, Railway, and Datadog ship agent-written code with full confidence because they know it's been verified by Antithesis. And to give a sense of how Antithesis help with large production systems, Datadog use Antithesis to verify their new architecture when rewriting the Datadog Event Platform, a system that collects more than 100 trillion events per day. Today, Antithesis is one of many correction strategies being used as part of Datadog's harness for their coding agents. To see more case studies and details, head to antithesis.com/pragmatic. Maude, welcome to the podcast.

Maude Lemaire 3:12 Thank you for having me.

Gergely Orosz 3:14 It is so nice to have

Gergely Orosz 3:15 you here.

Maude Lemaire 3:15 This is a bucket list item. Very much so. [laughs] I'm very-

Gergely Orosz 3:18 Well, this, this is the first in the podcast. Usually I ask, start with a question. I'm not gonna start with a question. I will show you a photo, and I will ask, "What am I seeing here?"

Maude Lemaire 3:28 [laughs]

Gergely Orosz 3:29 I'm seeing a baby-

Maude Lemaire 3:31 Yes, that's me

Gergely Orosz 3:31 ... uh, who, who looks like how many months?

Maude Lemaire 3:33 Maybe about a year old.

Gergely Orosz 3:35 A year old.

Maude Lemaire 3:35 Maybe a little before a year old, yeah.

Gergely Orosz 3:37 Sitting in front of a computer. Uh, we're, we're also putting this on, on the screen if you're watching it on video.

Maude Lemaire 3:42 Same haircut, so that's good. [laughs]

Gergely Orosz 3:42 Same haircut. So my, my, my, my question is, were you born with computers? [laughs]

Maude Lemaire 3:49 [laughs] Um, very nearly. There are pictures of me much younger than that, actually. Maybe, like, a couple months old. I, I truly... Like, I'm barely keeping my head up, where my dad has me sitting in his lap, like resting against him, and he's... I don't know what he's doing. He's probably, like, grading some kind of assignment or, um, because he, he was teaching. He, he taught computer science for 17 years before we moved to the US.

Gergely Orosz 4:15 Mm-hmm.

Maude Lemaire 4:15 And, uh, yeah. There, there's a bunch of pictures of me, like, holding ASCII tables. [laughs]

Gergely Orosz 4:21 [laughs]

Maude Lemaire 4:22 Um, which now in hindsight is hilarious. Um, but I ended up doing this sort of by accident. Um, you'd think that all of the variables and the environment would have led to, like, a very logical sort of linear career trajectory, and that is not at all what happened. [laughs] So anyway, he had to think about that, like, eons before. Um, but I thought it sounded really boring. Um, [laughs] I was like, "Uh, no, no, no." And I really liked my bio classes. I thought I was going to become a genetics researcher. That was my plan.

Gergely Orosz 4:53 Mm-hmm.

Maude Lemaire 4:54 And my parents always told me to chill the heck out because I had plans. I had... When I started university, I started in microbiology. I signed up for molecular biology, organic chemistry, all the things. I was already, I'd already accrued enough AP credits, for all the Americans listening, that I was technically, like, a second-year college student rather than a first year. Oh, yeah. [laughs]

Gergely Orosz 5:15 Oh, wow.

Maude Lemaire 5:16 Rather than a first-year college student. Um-

Gergely Orosz 5:18 Okay. So you, you, you want to speed run college.

Maude Lemaire 5:21 Oh, yeah. I was ready.

Gergely Orosz 5:22 Skip the fun years.

Maude Lemaire 5:23 [laughs] Yeah, which-

Gergely Orosz 5:24 Get straight to work

Maude Lemaire 5:25 ... which I kept doing that. I had, like, a 19 credit semester at some point, and I was like, "Why did I do that? That was such a poor decision." But, I mean, I had fun doing it, so I guess, you know, my, my brand of fun is, I guess, different from other people's brand of fun. And yeah, I was gonna do that, and then, like, three weeks in, there were a couple things that really bothered me. One, I... It's not because you're good at something that you like doing it, and I think I conflated the two for a really long time. I mean, to be fair, when you're 18, like, who... Like, what do you know? But I conflated being really good at memorization with liking memorization, and I don't know if you've ever taken a molecular biology class, but it is a significant amount of memorization. [laughs] And you're learning the Krebs cycle. So I was gonna switch into organic chemistry, into, um, uh, chemical engineering. And to do that, that's a different faculty at McGill. I mean, it's a very big school. There's a faculty of science. I was in the faculty of science. There's a faculty of engineering. And I was looking at all of the requirements for first year engineering students, and there was a program- programming class. There was a intro to computer science. So I swapped out one of my bio courses the next semester with the intro to computer science class.

Gergely Orosz 6:33 Computer science, yep.

Maude Lemaire 6:34 And, um, I told my mom, "I swapped it out for a computer science class. Please do not tell Dad. Please do not tell him that I swapped it out for a computer science..." Like, I don't... I'm scared I'm not going to like it, and then he's going to be sad [laughs] because he loves this so much.

Gergely Orosz 6:48 Yeah, he loves it.

Maude Lemaire 6:49 [laughs] This is, like, true love for him. And she was like, "Sure, okay." I'm pretty sure she told him, like, two minutes later, um [laughs] 'cause that's what my mom does. I love her, but, like, man. She, I think she knew that it was not going to be, you know, a wreck. And sure enough, eventually, like, I, I tell him I'm taking this class, and three weeks in I'm like, "Oh, I really like this." [laughs] Complete opposite from the molecular biology class, where I was like, "Ugh." I really liked it. It was like, oh, with the power... I'm sure you've had this feeling. It's like I only have a computer, and I type into it, and then it does things. I feel like a god.

Gergely Orosz 7:29 And then your first job, you did internships, but

Gergely Orosz 7:32 your first job, which followed the inter- was at Rent the Runway, right? Of-

Maude Lemaire 7:35 It was, yeah.

Gergely Orosz 7:36 It's a pretty well-known company.

Maude Lemaire 7:37 [laughs]

Gergely Orosz 7:37 Thanks to, thanks to anyone who's into your management will probably know of the book The Manager's Path by Camille Fournier, and she was at Rent the Runway.

Maude Lemaire 7:46 She was, yeah. And actually, what, that was a big part of why I wanted to, to go back after my internship there was because I... You didn't see very many women in those CTO-type positions at the time. Also, like, I mean, I, my mom very nearly became a fashion designer, and so we grew up watching Project Runway from the very beginning. That was, like, the one thing I was allowed to, like, stay up late to watch [laughs] was Project Runway. And so it combined two things that I loved. Uh, it combined, you know, programming, computer science, I was mostly doing front end stuff at the time, and, and fashion. I could just, like, rent stuff for dirt cheap and, like, walk around in a Monique Lhuillier, like, ballgown if I wanted to. It was, it was so cool. I think the, there were so many women on the team in general, I think partly because of the space that the company was in. Um, but I also just, for folks who've had the opportunity to be in a room with Camille, she just commands this presence, and I had so much respect for that. And I, I, I mean, even if, you know, I was many levels removed from her, I was like, "I wanna learn from this woman." Like, I really admire this, uh, this presence that, that she has and this confidence. And so, yeah, it was an, it was an easy sell. It was a [laughs] very easy sell to go back. But then she left a couple months [laughs] after I came back. But she was still instrumental in, um, in helping me land a job at Slack a year later.

Gergely Orosz 9:16 This is Slack in 2016.

Maude Lemaire 9:17 Yes.

Gergely Orosz 9:17 So at the time, I, I vividly remember because in 2016, I joined Uber. We still had HipChat, which was Atlassian's-

Maude Lemaire 9:25 Oh, yeah

Gergely Orosz 9:25 ... Slack offering, uh, or, like, Slack competitor. It was used by a lot of companies, and then they suffered two terrible and undefensible, uh, data leaks. Uber almost immediately migrated off to a product, and I remember that they evaluated Slack, and Slack would not scales. Uh, and I, I, I-

Maude Lemaire 9:43 [laughs] I'm not surprised.

Gergely Orosz 9:44 I, I, I thought this was, uh, later in, in 2019 Uber would move to Slack, but I wrote a th- a thread about this on, on Twitter, and I said, like, "Well, uh, we, we cr- we built something called UChat," which was built on top of Mattermost.

Maude Lemaire 9:57 Yes. Yes, yes, yes.

Gergely Orosz 9:58 And I was thinking like, "Hmm, did our in-house team just want to build something, and did they just, like, say that Slack didn't scale?" But one of the engineers who built UChat, 'cause they, they were working hand-in-hand with Mattermost. They were actually expanding it, uh, the open source offering. He actually said that they sat down with Slack, and Slack said that they cannot support 50,000 or 40,000 people in one room at the time. So there, there was no way. So, like, just to set the stage, in 2016, Slack still maybe struggles to serve more than 10,000 people in one room if, if I remember.

Maude Lemaire 10:30 I think it was about 30.

Gergely Orosz 10:31 Or 20,000 or 30,000.

Maude Lemaire 10:32 I think 30,000-

Gergely Orosz 10:33 Or, or-

Maude Lemaire 10:33 ... was Comcast. 30,000.

Gergely Orosz 10:36 But, but then-

Maude Lemaire 10:37 It was like... Yeah

Gergely Orosz 10:37 ... they can do be- be- 40,000, so this is the state-

Maude Lemaire 10:40 Yeah. Oh, yeah, yeah.

Gergely Orosz 10:40 This is the state of Slack. This is early Slack.

Maude Lemaire 10:42 Early.

Gergely Orosz 10:42 It was, it was already v- v- wildly popular. People loved the, the onboarding. Teams was in infancy. I'm not even sure if it was born at the time. So-

Maude Lemaire 10:51 Yeah, I don't think it was

Gergely Orosz 10:52 ... you, you, you get to

Gergely Orosz 10:54 Slack.

Maude Lemaire 10:54 Yeah, yeah.

Gergely Orosz 10:55 And what, what were the problems you had to work on?

Maude Lemaire 10:58 Oh, gosh.

Gergely Orosz 10:59 That, that, that you got to work.

Maude Lemaire 11:00 Yes. Yeah, um, yeah, I very much got the, the performance itch from, from Rent the Runway. It just wasn't, I just had to work on it, you know, on my own time, but I still enjoyed doing it. And then when I showed up at Slack, um, I, my onboarding buddy was, um, this, like, very senior engineer who was working on performance issues specifically at the time. And I burned through some of, you know, my starter stuff, and I was like, "What are you working on? What are you working on over there, Moisha?" And he was like, "Oh, let me send you some tickets. Like, do you wanna work on performance?" I was like, "Yes. Yes, I do." [laughs] So he sent me a couple things, um, like, relatively easy stuff. In startup land, when you need to get to the next place and just get that next customer or build that next feature that you think is gonna capture, like, you make a lot of short-term decisions that you know probably inside your soul you will revisit at some point. Um, and a lot of the performance work that we were doing at the time w- was just a lot of revisiting of shortcuts. Um, I mean, like, well-calculated shortcuts, but, like, shortcuts that we knew were gonna, like, bite us in the butt eventually down the line. So a lot of that, um, work I was doing, and this was before the, like, Enterprise, the Slack Enterprise product was launched. So I started a couple months before GA. IBM was already trialing The product

Gergely Orosz 12:22 Mm-hmm

Maude Lemaire 12:22 They were sort of on beta there. Um, [laughs] they, they are a large company. [laughs] Lots of people work there

Gergely Orosz 12:28 Yeah, they have, uh, less than 100K-plus people, if I recall.

Maude Lemaire 12:32 Yeah. So one of the distinctions, actually, was, um, it was sort of fine to have tens of thousands of folks on one... in one channel at one time

Gergely Orosz 12:42 On one channel, yeah

Maude Lemaire 12:43 Um, and we were sort of spreading them out across different teams within the same Slack organization. Um, obviously, they could DM each other, amongst some other things, but that sor- sort of worked. Uh, but it was... IBM was sort of falling over on a regular basis. And I think, I was not part of these conversations, but I get the impression in hindsight that when we agreed to GA, then they agreed that they would start scaling and adding more users to Slack. Like, once we reach GA, you know, we, we can't hold the reins on you adding more people as much. Um, so we hit GA, and they were started adding more people, like, pretty aggressively. Uh, we started-- we ended up having to spin an actual team up that was just the enterprise performance team. I ended up being able to join that team 'cause I just happened to be working on odd random performance things. Um, [laughs] I was definitely the most junior person on the team. But I love that. I feel like I've spent so much of my career to date being the most junior person on the team. I mean, less so in the last couple years, but I-- that's how I've learned the most, and it's really just, like, pushed me out of my comfort zone over and over again, and it's just like, it's the most, it's the most fun.

Gergely Orosz 13:58 And can we do a bit of a Slack arc in-

Maude Lemaire 14:01 Yes

Gergely Orosz 14:01 ... the, the 2016, 2017 Slack architecture 101, 101. Now, this is, as we know, Slack has changed a lot. This is now how, not how it will look like today-

Maude Lemaire 14:09 Mm-hmm

Gergely Orosz 14:10 ... at all. But when we think about Slack workspaces, then channels, private groups, DMs, behind the scenes, how was it built up in terms of databases, in terms of what happened when you sent a, a message to someone or to a channel in terms of fanouts or writes?

Maude Lemaire 14:30 So the pattern before Enterprise, and still with Enterprise, was everything belonged to a company, a workspace, a team unit, and all of the data also was sharded by team. And so if you were on a really big team, well, we just would give you a bigger database. [laughs] And eventually, uh, you run out of the largest size of database that you can throw money at Amazon for. [laughs] Like-

Gergely Orosz 14:59 But, but, but you're saying there, there was a time where this sharding worked where a large company might just have its own database instance, and-

Maude Lemaire 15:05 Yes

Gergely Orosz 15:05 ... for, for, for a while, they, they-

Maude Lemaire 15:06 It would work

Gergely Orosz 15:07 ... it would work. They didn't fill it up.

Maude Lemaire 15:08 Yeah.

Gergely Orosz 15:09 And smaller companies would share. They would be multiple on the same-

Maude Lemaire 15:12 Multi-tenant on the same-

Gergely Orosz 15:13 ... multi-tenant d- database

Maude Lemaire 15:14 ... uh, database instance. But you as a customer, your data was not spread across multiple database shards. It was just on this one. Um, it was also how we distributed messages. So for the WebSocket connection, when you were connecting, logging into Slack, establishing that WebSocket connection, you were establishing that WebSocket connection with a server that was associated with your company. And so if everyone at your company was very busy, very busy chatting on Slack, you all were on that one same server. And again, like, if we had really, really busy customers, eventually we could just say, like, "Oh, well, this is your instance of the chat server."

Gergely Orosz 15:53 Yeah.

Maude Lemaire 15:54 Um, and that was mostly fine for a very long time. Enterprise started complicating that, um, in some special ways in that we... I mean, again, startups, you, like, sort of make decisions that are fine in the moment, but then sort of don't scale. We sort of made the assumption that, like, inside really big companies, it was smaller companies.

Gergely Orosz 16:16 Mm-hmm.

Maude Lemaire 16:17 And so, like, marketing at, say, IBM is its own company, so you sign into the marketing workspace at IBM, or the engineering works- I mean, engineering actually is too big. It's some subset of engineering at IBM.

Gergely Orosz 16:31 Oh, it, so it, it sounds like it's a bit of wishful thinking that you could just turn this large problem into smaller problems, right?

Maude Lemaire 16:37 [laughs] Yes.

Gergely Orosz 16:37 Which-- and then you can just make your current architecture work.

Maude Lemaire 16:40 But there still needed to be, like, an overarching body, um, so we created an Enterprise which was really just [laughs] another row in the team table that just had, it is enterprise one, and, [laughs] and then all of the teams associated with that enterprise had an enterprise ID. And you still, within IBM, would want to say, you know, say you were working on one of the engineering teams. I was working on a different engineering team. We wanna collaborate. I wanna be able to DM you.

Gergely Orosz 17:07 Yeah.

Maude Lemaire 17:07 We're at the same darn company. I should be able to send you a message. We pretended [laughs] that these teams within a company were their own companies, but then we had to create this other entity that overarched everything where you could still cross teams, and that got very complicated very fast. But then the part that became really terrible was everyone across all of IBM, whether they were on the marketing team, the engineering team, whatever, were all querying the enterprise shard for all of their DMs and the big company-wide channels, all of that stuff, all at the same time. And then within that data, when you're booting up Slack, you have your channels that show up in the left sidebar. You'd think that it was just like, cool, get, you know, for user ID, get all channel membership entries. Um, that was not the case. [laughs] And I think it came from a good place in that DMs seemed like a separate thing from a regular public channel, right? Like, you-

Gergely Orosz 18:07 Yeah

Maude Lemaire 18:08 ... th- they seem different. Um, and so that was a separate table. And a public channel is a public channel.

Gergely Orosz 18:13 Yep.

Maude Lemaire 18:13 And so that was a separate table. But then private channels seemed different. And I think, like, from a product perspective, you can totally see it, and from a product security perspective, you can also totally see that. You want more scrutiny when you are querying for private channels.

Gergely Orosz 18:27 Yep.

Maude Lemaire 18:28 And so that was a separate table as well. And then group DMs came about later, and it seemed like they were just sort of like a private channel, and so they were just glued into the private channel table. So when you were making that query to populate your sidebar, you were actually doing a big, big old join union.

Gergely Orosz 18:44 So what happened once, once you and... Once the team had the idea or f- figured out that this, these tables don't make sense, but now you're talking about a lot of customers, including the likes of IBM, just, you know, putting a bunch of load on it. They're using it.

Maude Lemaire 18:58 Yeah.

Gergely Orosz 18:58 How do you migrate from these-

Maude Lemaire 19:00 [laughs]

Gergely Orosz 19:00 ... three tables that are actively being used into one?

Maude Lemaire 19:03 All the time. Yeah. I think it came about from calculating mentions. So calculating mentions, like, to bad- get the badge icon, uh, was also a very complex query whereby you had to join across all these tables, um, very, uh, latency-sensitive. And we had gone through the process of doing as much as we possibly could at that time to ease the load. And in one of my teammates doing some work on mentions, he was like, "We gotta c- we gotta join these tables together. It will not... Like, there's only so much we can do before we have to, like, really start digging the hole." So I, I was like, "Sure, yeah. I, I will endeavor to do this very scary thing that no one else wants to do. That seems totally fine." I have so much appreciation for this in hindsight too, in that if you look at Flamework on GitHub, which is the PHP framework that Cal Henderson open-sourced after writing Flickr-

Gergely Orosz 20:04 Mm-hmm

Maude Lemaire 20:04 ... it will explain many things about what Slack was like early on.

Gergely Orosz 20:09 Hm.

Maude Lemaire 20:09 Yes, I highly recommend. It's, it's a good... I, I, I think it's still there. [laughs]

Gergely Orosz 20:13 Oh, okay.

Maude Lemaire 20:13 It's still there. It's a g- it's a good read in terms of like, "Oh, this explains so many things." So we had, everything was super procedural, for lack of, of better term, in terms of how code was organized. Um, we had sort of like API entry points, and it was like chat.login or whatever, right? And it was like, "Okay, this is the API, and this is all the code for the API with some helper functions in there." There were a wild amount of functions that were, like, many, many, many, many, many lines long. But there was just, like, straight SQL queries, like, embedded inside the code. Like, an ORM did not exist in this world.

Gergely Orosz 20:49 Yeah.

Maude Lemaire 20:50 Uh, whatever you want to call your, the data access layer, we did not have one. And there were actually, um... There was a lot of opinions about whether we should even do anything to hide the SQL queries. And yeah, now, I mean, I totally understand why, why they had that stance and why we did the super minimal approach that we ended up doing because it is so easy to write a SQL query that will, like, shoot you in the foot over and over and over again, unless you're forced to actually understand what it does in the context that it is being called. But we had like 400 something call sites that called into any number of these channel tables.

Gergely Orosz 21:34 And, and when you say call sites, these are SQL instructions in the code, right?

Maude Lemaire 21:38 Just like select star from-

Gergely Orosz 21:41 Yep

Maude Lemaire 21:41 ... teams, channels where user ID blank.

Gergely Orosz 21:44 Yeah.

Maude Lemaire 21:44 Where team ID blank, user ID blank. And so you just copy-paste the pattern that exists, right? That's always the problem with migrations is like you have to make it so clear that, like, "This is the old pattern. Stop doing this."

Gergely Orosz 21:55 Yeah.

Maude Lemaire 21:56 Because there are still, while it's going on, so many examples of it in the code. So I had to write some... I started off by going through, doing some kind of regex approach where I was finding all the places that we were querying for this. Thankfully, the table names were, like, pretty easy to find and identify. Got very lucky there. [laughs] And then I just one for one took the call and just pasted it into another file.

Gergely Orosz 22:19 Yep.

Maude Lemaire 22:19 That is all I did.

Gergely Orosz 22:20 You, you created a layer that did the same thing.

Maude Lemaire 22:22 Yeah.

Gergely Orosz 22:22 Just a class, basically.

Maude Lemaire 22:24 Yeah, yeah. I didn't optimize anything.

Gergely Orosz 22:26 Yeah.

Maude Lemaire 22:26 I literally just copy-pasted, and now it was calling into a function instead of straight up calling SQL.

Gergely Orosz 22:31 And you had a minimal, but ab- abstraction layer, if you will.

Maude Lemaire 22:35 Yeah, which already was, like, a little bit spicy. Um, but I tried to make sure that, like, the function name was, like, real darn clear as to what it was doing. Um, and I got the spicier folks to be okay with this approach, so I felt, that felt like a win. And so I migrated everything over. Everything was feature flagged. Basically, I had, I don't know how many feature flags at some point, but I had so many distinct ones just to make sure that I was doing it as carefully as humanly possible. Even then, you know, I still, like, definitely borked some things along [laughs] the way. I accidentally, when I thought that we were basically there and ready to start flipping on to prod traffic, I'd run all the backfills because obviously, like, I was moving everything to a new table. I was doing, um... We were doing, uh, double writes for a very long time.

Gergely Orosz 23:25 For a while, yep, yep.

Maude Lemaire 23:26 While we were-

Gergely Orosz 23:26 Writing to both places to make sure the data will be up to date.

Maude Lemaire 23:30 Yes, so that's where we started. We started doing double writes, making sure that that all looked sensible. I was doing a bunch of-

Gergely Orosz 23:35 Did, did you also do, also do shadowing?

Maude Lemaire 23:38 Yes.

Gergely Orosz 23:39 Shadowing meaning that you're, you know, like you're doing whatever you're doing and you're doing new operation and comparing that as the same result, right?

Maude Lemaire 23:46 Yeah, yeah. We did a bunch of that as well. Um, but obviously we had to, you know, make sure all the data was there first. So we, like, did the double writing, did all the backfills, um, and then we were shadow reading and comparing if there were any differences. Throwing away the new, uh, the new call-

Gergely Orosz 24:00 Results

Maude Lemaire 24:00 ... obviously, the new results, and, um, and keeping reading from the old ones. I was about to flip all of Slack to reading from the new table.

Gergely Orosz 24:11 Which at this point you confirmed worked, so that should've been a low-risk change.

Maude Lemaire 24:14 Yes. Yeah, yeah, yeah. All the data looked right. I had a bunch of data warehouse dashboards that showed me how many differences existed between the two sets of tables within reason. And, uh, and then I was about to turn it on to, um, to all of Slack. This was like Thursdays back when we had gather hour where it was like after, after work hours, everyone would sort of hang out if they wanted, have appetizers, whatever. And I turned it on, and then I realized, oh wait, no, something's not quite right. Um, and [laughs]

Gergely Orosz 24:42 Oh, oh, and then, and w- we didn't just do a gradual rollout 'cause with these things you don't really do a gradual rollout.

Maude Lemaire 24:47 Internally We, mm, we were, like, not that big at the time. Everyone was at gather hour. You know, I was like, "Oh, this seems safe. We've been shadow reading, it's fine." Yeah, I turned it on, I was like, "Oh, there's, like, data isn't... There's some data that's, like, not quite right." Um, and I had been in so used to, uh, essentially, like, running this script that, like, would delete all of the backing data and rerun the backfill, or at least chunks of it, because I had been seeing discrepancies, and honestly, it was so much easier to just nuke the data, re- re-backfill, than it was to-

Gergely Orosz 25:23 Yeah

Maude Lemaire 25:23 ... try to, like, pinpoint and-

Gergely Orosz 25:25 Yeah

Maude Lemaire 25:25 ... like, scalpels fix the, the call sites. So I forgot that the flag was still on to everyone, and then I promptly deleted-

Gergely Orosz 25:34 Which flag? Which flag was on?

Maude Lemaire 25:35 Uh, the flag to have everyone read their channel memberships from the new table.

Gergely Orosz 25:39 Okay.

Maude Lemaire 25:39 Yeah. I forgot that it was on for Slack. Not everyone externally-

Gergely Orosz 25:43 Yeah

Maude Lemaire 25:43 ... like, just us.

Gergely Orosz 25:44 Yeah.

Maude Lemaire 25:45 And I was like, "Oh, there's some discrepancies. Oh, right, but I fixed those. Let me just nuke the table for our team and backfill," which, like, took 25 minutes to do. Like, it was not-

Gergely Orosz 25:56 Oh, but ev- did everyone lose their messages until then?

Maude Lemaire 25:59 Oh yeah, people had completely blank sidebars for about half an hour.

Gergely Orosz 26:03 In- in- inside of all of those. Like-

Maude Lemaire 26:04 Inside all of Slack

Gergely Orosz 26:05 ... so, so like, and Slack obviously uses Slack for everything.

Maude Lemaire 26:08 For everything. But everyone was at gather hour, so it, like, there was only [laughs] there was this only one guy, Kevin, who had been sort of, like, help- helping me, like, coach through this, who DM'd me and he was like, "What's going on with my sidebar?" [laughs] I was like, "I'm so sorry. Oh my gosh, let me flip the flag." [laughs] So I re-flipped to read from the old data, um, while it was backfilling. It finished backfilling and then I swapped it back.

Gergely Orosz 26:29 So

Gergely Orosz 26:30 what did you learn at Slack about load testing? And I'm asking because right now load is going up on all systems-

Maude Lemaire 26:36 Oh my gosh, yes

Gergely Orosz 26:37 ... of, you- we, we see this bas- as a- AI agents generate more pull requests, uh, more code, uh, they put more load on the systems. I assume we will have the point where we generate more data here and there. But at Slack you, you had this interesting challenge where a customer with 500,000 users wanted to go onto the platform, Slack was architecting for it, and you had a plan of 18 months, and then COVID happened and you had to bring it up forward. But I- I'm more interested in that, and as you, you built load testing of something called a koi pond on this. But when you think of load testing, of checking how a service will be able to handle this type of load before it hits, what are the techniques that you've used and would still use today?

Maude Lemaire 27:22 Test in production. [laughs]

Gergely Orosz 27:24 But how, how to test in production something where you don't have the load just yet, right?

Maude Lemaire 27:28 Yeah, no, I mean, I think there's a lot of different ways to do it. The simplest way that I think a lot of people can manage to do, it, it depends on what kind of usage you are worried about with your product. Um, and nailing down those usage patterns first, and where you're really concerned it's going to fall over, is the determining factor in how you figure out how to simulate load.

Gergely Orosz 27:52 In Slack specifically, what was usage pattern?

Maude Lemaire 27:55 There were a couple, couple key ways that it had historically fallen over. Thundering herds, where, I mean, I think most people have some kind of thundering herd problem at some point, where you, say, have-

Gergely Orosz 28:05 Can, can, can we spell out what the thundering herd is?

Maude Lemaire 28:07 Yeah, yeah

Gergely Orosz 28:07 ... just for those who have-

Maude Lemaire 28:08 Oh, yes, yes, yes

Gergely Orosz 28:08 ... who have, who have not yet had a herd

Maude Lemaire 28:10 ... thundering herd is, like, suddenly herd [laughs] suddenly everyone shows up. Suddenly, like, that, that is a thundering herd. A thing happens and then suddenly everyone shows up and does the same thing at the same time.

Gergely Orosz 28:20 And I, I guess a good example is if, if your service goes down, like your, I don't know, you're running, let's say, imaginary, like, Google search and somehow, like, your, all your machines die for a bug, and then you, you have, like, I don't know, 1,000 machines. Obviously Google has more, but we can just imagine. Uh, and then you start to bring up machine one, machine two, machine three, but each of them gets hit by the complete traffic and they go down again, right?

Maude Lemaire 28:43 Yes. Yes.

Gergely Orosz 28:44 Like, this is a classic example.

Maude Lemaire 28:45 Oh, yeah. Yeah. The classic example in Slack is, say, CEO says, "Oh, great quarter everyone. You all get Friday off," posts this message in the big company channel that has, like, 100,000 active users in it. That notification gets sent out. You see that notification. You are very motivated to plop a really good custom emoji on that notification. You click that notification. You immediately land in the channel. In the channel, you're fetching channel history. And then on top of that, you are adding an emoji reaction because you're really pumped about getting Friday off. But so is everyone else also getting, uh, putting custom emojis on that message-

Gergely Orosz 29:22 In the-

Maude Lemaire 29:22 ... because they're also pumped

Gergely Orosz 29:23 ... in, in this, in the same five seconds give or law.

Maude Lemaire 29:25 Yes, in the same five seconds. So the message that came in to tell you that you're getting Friday off came in over the WebSocket connection, and it was fanned out to all 100,000 active users. But then all 100,000, within a couple seconds, right, of those people show up to that channel, issue a channel read, and then are sending emoji reactions. Every single emoji reaction fans out also over the WebSocket. So one custom emoji turns into 100,000 custom emojis being sent to everyone who's connected, and if you add 10, well, then you can do the math, and it just, like, balloons and balloons and balloons. It's like prime example for, like, pure explosion. And, um, as much as... I, and I think we, we had done some work because we knew that emoji fan out was, like, actually really expensive to rate the importance priority order, the types of messages that get sent over the WebSocket. And so, like, mess- like, actual text messages were the most important, and then emojis, if, uh, you were in a very high load situation, would just get dropped on the floor, which, like, is the thing to do, obviously.

Gergely Orosz 30:29 And I'm, I'm, I'm sorry to say, but the next generation, well, I'm not sure if Gen Z or Gen Alpha might disagree that emojis are more important than text messages.

Maude Lemaire 30:36 [laughs] More important. I need to know if someone reacted to my message. Um, [laughs]

Gergely Orosz 30:42 No, but-

Maude Lemaire 30:43 Yeah, yeah, yeah

Gergely Orosz 30:43 ... I, I, I hear you. It makes sense with prioritizing. Es- especially based on size, right?

Maude Lemaire 30:48 Yeah. Yes, absolutely. And there's some amount of, and this is where, like, I think product thinking is you can't go around thinking you're gonna load test strictly from an engineering perspective. You have to be putting your product hat on. You absolutely have to because that's where Understanding that, like, messages with text are more important than emojis, that's more of a product call than it is, like, an engineering call. I'm coming to you and I'm saying like, "We have to drop something. Let's reason together about what makes sense to drop." Um, and understand it, and sometimes it's, like, not what you would expect. And I think, I, I mean, I, I... This is an example I keep coming back to because I had multiple conversations with folks over at Slack about it. Like, we should just limit the total number of emojis that are allowed on a message, period. So we currently limit, or s- I think the last I checked, Slack still currently limits, you can only have 23 distinct custom emojis as a reaction on a message, but you can have as many, like, plus ones on those emojis as you would like. Why? The 10,001th emoji does not matter. We should just cut it off [laughs] at some point. And I think, like, it- whoever ... I mean, I think part of the problem is, like, no one really owned custom emojis. It would've been the messaging team, I guess. But, like, it's one of those things that's just so ubiquitous. Is that, is that a word? [laughs]

Gergely Orosz 32:10 Yeah.

Maude Lemaire 32:10 To the Slack experience, and, like, kind, kind of no one owns it. Um, so who would've made the call on, like, what is the proper limit to put on this? But thundering herds was definitely super top of mind as one of those issues.

Gergely Orosz 32:23 I wanted to ask, are event floods and thundering herds the same thing?

Maude Lemaire 32:28 No.

Gergely Orosz 32:28 Or-

Maude Lemaire 32:28 They're

Maude Lemaire 32:29 different

Gergely Orosz 32:29 ... w- what are, what are event floods?

Maude Lemaire 32:31 So event floods are when your client is personally overwhelmed by the events coming over the WebSocket connection. And so any, I mean, that could be from that specific example, that sort of thundering herd and maybe an event flood event in that you're getting hammered with emoji reactions after the fact as well. Um, but an event flood is, uh, just, say, you know, this message is happening, the CO you get Friday off, but it's happening in 16 different channels at the same time that you personally happen to be in. Not everyone else is in super, super busy channels, but maybe you happen to be. This happened to, there was this guy who was our, like, main Slack, Slack guy at IBM. His name is Jeff Robke. He eventually became a PM at Slack, which I think is, like, the most interesting thing ever because this guy was on the receiving end of so many event floods, and I felt so bad for Jeff all the time. I mean, I was in some channels, I'm like, "I'm"... Like, "Tell Jeff I'm sorry." [laughs]

Gergely Orosz 33:34 But I, I, I still don't fully get, like, what the event flood experience is. Can you tell me, like, what, what, what did, what Jeff experience, what example-

Maude Lemaire 33:38 Yeah, yeah. So, so one of the things that would happen to him, right, is that before some of the rearchitecture work whereby you didn't have a bunch of distinct WebSocket connections for each team that you were logged in-

Gergely Orosz 33:51 Yeah

Maude Lemaire 33:51 ... Jeff was an admin at IBM, so he was in hundreds of different workspaces.

Gergely Orosz 33:58 Yep.

Maude Lemaire 33:58 At one point, I think the maximum number of workspaces you could be logged into was around, like, 30-something.

Gergely Orosz 34:03 And, and you're saying he was in this because, uh, IBM as an enterprise had this i- this concept of, like, the marketing team, the finance team, the thing. Maybe they had an external customer team and as the admin he was in all of-

Maude Lemaire 34:14 He was in all of them.

Gergely Orosz 34:16 Yep.

Maude Lemaire 34:16 And they had so many teams.

Gergely Orosz 34:19 Yep.

Maude Lemaire 34:19 And he was in all of them, and so if you were getting a message in an enterprise level DM or a channel that is shared across all 30 of those workspaces that he was currently logged into, well, he would get that message 30 times. There was no deduplication of that whatsoever.

Gergely Orosz 34:36 So his client, his machine, his notifications would be both overwhelmed from m- m- maybe a little bit less in processing, that's fine. But, like, from a UI, from a, from a being un- un- unable to, to see what's going on anno- annoyance point of view.

Maude Lemaire 34:52 Yeah, I mean, also his client was just dying all the time. [laughs]

Gergely Orosz 34:56 Yeah.

Maude Lemaire 34:56 Which was not good. [laughs]

Gergely Orosz 34:57 Yeah.

Maude Lemaire 34:58 Yeah. [laughs]

Gergely Orosz 34:59 And interesting, like, I, this is the first time I've heard about event floods.

Maude Lemaire 35:02 Yeah. And, and they were very user specific, um, but they were particularly painful for power users. This happened to power users all the time, and power users were also some of the, like, biggest champions of Slack. And so [laughs] it was a double-edged sword of, like, I'm sorry you get the absolute worst experience of our product, but also you love it, I guess, and you are trying to, like, make sure that people adopt it more. [laughs] It's really, really rough.

Gergely Orosz 35:26 Maude just made the point that the heaviest users of Slack got the worst of the event floods. Today, the heaviest users of Git happen to be your agents that often push code in parallel, which brings us to our season sponsor, Entire. A few weeks ago, I told you that Entire was building a Git forge. Well, it's finally here. You can now host private repos directly on Entire in the US, Europe, India, and Australia. And Entire has just reinvented the pull request for the agentic era. The problem with PRs is how agents write it faster than how any of us can review it, so Entire just replaced it with something called trails. A trail collects everything about the work in one place: the intent, the code changes, and the agent sessions. A trail can be spread across multiple repos, so the change to your backend and your web app gets reviewed once. With a trail, you can configure customized evals like code review, security reviews, slop reviews, and others. They look like a heads-up display on an airplane and score your work on every push. And you can see how each agent from passes or fails your user-defined gates and then decide when you merge based on these. The point is that you control when you allow your agents to be allowed to merge code, like merging only after human review or after all gates pass. Oh, and one more thing. Entire ship continuous memory. This turns your repo into a brain so your agents can search past coding sessions to find the context behind earlier changes. Basically, answer the why behind the code changes. Memories get shared across the team and across the repo. If you want to see what an AI native Git forge looks like, head to entire.io/pragmatic and check out trails. I'd also like to mention our presenting sponsor, Turbopuffer, who are rebuilding their storage engine to make search faster and cheaper at the kind of scale we're talking about in this episode. The new system, called Turbopuffer V3, is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version, V2. The T-puff team has been perf maxing query plans until they are as fast or faster than V2, doing all of this in public all the way up to the V3 production launch. And it's so cool to see their progress. The new system went from being 10 to 100 times slower across most metrics to several times faster on things like cold vector search, cold full-text search, or counts. The V3 system is still slower in areas like hot full-text search, but only one to two times slower versus 100 times slower when the perf maxing started. And the T-puff team is sharing exactly how they optimize things in their development log with things like vectorized query execution, group count aggregation, compressed column encodings, and many more. But hang on. Why is the team doing a full rewrite? It's because search is changing and so is TurboBuffer. T-puff started as a vector database, but it's no longer a vector DB. It has evolved into a high-performance database for search. The V3 discards the vector primary index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something that you don't usually see in search databases, but which will both result in very fast search and just generally a very fast database. The TurboBuffer team is keeping a work log of their journey, and you can follow along at turbopuffer.com/v3. It's a blast

Gergely Orosz 38:30 to see how it's being built. And with this, let's get back to mode and how Slack load tested for a customer expecting to have 500,000 users. And then back to load testing, so how did you simulate the load that you would, uh, be expecting to get? In this case, uh, you knew that a big customer would be coming with 500,000 or 1 million users.

Maude Lemaire 38:51 Yeah.

Gergely Orosz 38:51 And you didn't know if Slack could handle it, but you want to figure out before testing in production, before they all came in. Did you build a custom load testing tool? H- how did you go about it?

Maude Lemaire 39:00 Yeah. Um, so we, we'd had a couple of, like, half-conceived load testing options to date, um, mostly that tested just API throughput and making sure that we could withstand the underlying load primarily on the database. And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they've last changed it, and that makes a particular database request, uh, a SQL read on, on that channel shard now because everything is threaded differently, but it makes a particular request. We wanted to be able to... We wanted to know that we could handle that RPS. The problem with that is, and, and any of the load testing solutions before this particular moment in time, was none of it had active WebSocket connections. And the two, uh, particular cases that we just talked about, the WebSocket is the problem in some ways. Um, I mean, we've never really had latency issues with sending messages through the WebSocket. Like, those stateful services have-- One of the co-founders was, like, the mastermind behind all of these, and, like, the amount of time and energy he sank into making them as performant as humanly possible is commendable. They were never really a bottleneck. It was either how the client handled it or the read downstream event caused by you seeing and receiving this message that was the further bottleneck. And so we needed a system that had an active WebSocket connection in order to be able to adequately test the whole flow. It's not just sending, like-

Gergely Orosz 40:35 Mm-mm

Maude Lemaire 40:35 ... as though you were reading a channel.

Gergely Orosz 40:36 It's not just sending an API.

Maude Lemaire 40:37 Yeah.

Gergely Orosz 40:37 I get it.

Maude Lemaire 40:38 So we really-- And it's not just, like, flooding a database with, like-

Gergely Orosz 40:41 But when, when you say you need an active WebSocket connection to, to test for 500,000 load, you needed-

Maude Lemaire 40:46 Yeah

Gergely Orosz 40:46 ... 500,000 active web-

Maude Lemaire 40:47 We needed 500,000-

Gergely Orosz 40:48 ... thousand connections

Maude Lemaire 40:48 ... active WebSocket connections. Yes, we did.

Gergely Orosz 40:50 Okay.

Maude Lemaire 40:51 Yeah, yeah, which is not easy to do. [laughs] And so we had a solution before this, this customer came in, which I'm 90% sure it's public knowledge now that it's Amazon. But we had a solution that would spin up Slack as a headless Chrome instance in, like, a bunch of Kubernetes pods-

Gergely Orosz 41:09 Mm-hmm

Maude Lemaire 41:09 ... and try to run a tiny little JavaScript script, um, and t- like click buttons.

Gergely Orosz 41:17 Yeah.

Maude Lemaire 41:17 Like pretend to be-

Gergely Orosz 41:18 Pretend to be a-

Maude Lemaire 41:18 ... a Slack user

Gergely Orosz 41:19 ... the user, yeah.

Maude Lemaire 41:20 Which, like, was sort of fine. Um, but I don't know if you've tried to wrangle, um, more than, like, 50,000 headless Chrome instances in Kubernetes. [laughs]

Gergely Orosz 41:34 I, I don't think many of us have.

Maude Lemaire 41:35 [laughs]

Gergely Orosz 41:36 I, I don't think many people listening or viewing have. [laughs]

Maude Lemaire 41:40 It is not a good time. Um, I mean, the problem with the headless Chrome instance, right, is, like, just running Slack in, like, 200 Chrome tabs and whatever on your laptop is, like, kind of a time. Um, and now you're running, like, you know, LLMs on top of it. But, like, it's an experience in really pushing your laptop to the limit. And then [laughs] now you're running, like, per pod, like, attempting to run, like, 10,000 headless Chrome instances of Slack, which is already, like [laughs] a very memory and CPU, m- mostly memory-intensive, um, application. The, the amount of money we were spending in-

Gergely Orosz 42:13 I, I was about to say, like, uh-

Maude Lemaire 42:13 ... these EC2 instances. [laughs]

Gergely Orosz 42:15 The, the problem seems to be that per, per node, per, per VM node, depending on the size, you will be able to run, you know, you, you, you'll, you'll know, like, I don't know, like 10, 20, 50, 100. I don't know. You figure out how much. But then you pay for every one of those VMs, those nodes. You have a number of them.

Maude Lemaire 42:31 Yeah.

Gergely Orosz 42:31 Uh, I mean, look, like, Slack, Slack raised a bunch of VC money, so I guess-

Maude Lemaire 42:34 Sure, sure. Yeah.

Gergely Orosz 42:35 So we, we know where some of that went.

Maude Lemaire 42:37 Yeah. Yeah. But-

Gergely Orosz 42:37 And also from, from customers, of course.

Maude Lemaire 42:39 [laughs] Yeah. Yeah. Yeah. Yeah. Of course.

Gergely Orosz 42:40 But that, that is wicked expensive.

Maude Lemaire 42:42 It is so-- It was 37 cents per, uh, per headless Chrome instance per five-hour tranche of load testing.

Gergely Orosz 42:50 Yeah, we're talking about $150,000 for that one test.

Maude Lemaire 42:53 Mm-hmm. Yeah. Yeah, yeah. Which, like, okay, that's, like, financially not sensible. Like, we-- Like, it just didn't work.

Gergely Orosz 42:59 But, but, but I guess if, if you compare it, like, of course, I'm playing devil's advocate, but-

Maude Lemaire 43:05 Oh, yeah

Gergely Orosz 43:05 ... like $150,000, great, but if you have a customer coming on with 500,000 seats, if they were to get a discount, they, they would pay $5 million per month

43:14 Like, y- you would think there is, there is a point of investing to make sure that it works for them and they don't complain. I don't know where that comes in, but, but yes, the $150,000 still feels like pretty excessive for that one test.

Maude Lemaire 43:27 Oh, yeah, yeah. I think for, and for that one, I mean, that's only a five-hour chunk of time, and that's assuming that you're able... Yeah, that's only five hours.

Gergely Orosz 43:34 Okay.

Maude Lemaire 43:35 Yeah.

Gergely Orosz 43:35 That's gonna be a problem.

Maude Lemaire 43:37 And you're, that's assuming you're testing all of the feature tests that you want to run during that five hours.

Gergely Orosz 43:42 Which was not the case. [laughs]

Maude Lemaire 43:43 Which, [laughs] which again, that was not the case because we were spending half the time trying to bring pods back up because they were crashing.

Gergely Orosz 43:51 Okay. Did you figure out a better way to do it in the end?

Maude Lemaire 43:53 Yes. Yeah, we did. Um, lower fidelity, but definitely a better way. We gave up on the headl- whole headless Chrome instance situation, which was obviously like the highest fidelity and best option if you wanted to make sure that it was truly testing how Slack would work. Um, but instead we spun up a bunch of little Go processes that would each establish a WebSocket connection and run through a series of API requests that made it look like they were actual clients.

Gergely Orosz 44:21 Mm-hmm. Mm-hmm.

Maude Lemaire 44:22 Um, and so they would still receive everything over the WebSocket, send everything over the WebSocket as you would expect, and still have a connection to the regular API that the client was making requests to, and also our edge cache. We didn't talk about our edge cache at all, but there's a bunch of data that needs to be really snappy to load when you load up Slack. Like when you go to switch channels, the channel list is, um, is in an edge cache l- closer to where you are because all of the main API requests still hit US East One, which is great. I mean-

Gergely Orosz 44:53 Yeah

Maude Lemaire 44:53 ... we are so boned if US East One goes down. All those main requests still hit US East One, so we still had to also test the latency associated with making requests to, um, to those edge caches, and so we also established that request pattern there.

Gergely Orosz 45:08 It was, it sounds to me that with load testing, with, well, all, all, all I've heard from you is like, you know, step one is understand how your product will be used, what will likely top it over, which, uh, because once, once you know, you're not gonna just randomly load test. You'll actually can test target it for that, and then building something that simulates it good enough, knowing the resource constraints. So in this case, don't boot up a whole Slack in a headless Chrome, but just mock-- not mock up, but build out that critical path that you care about. And if you combine this to your domain knowledge and your expertise, knowing how the product works and knowing how, you know, you, you knew how it used to fall over, you can now be reasonably confident that if it falls over, it will not be this way. It might be a different way, but I think it's nice that we're talking about the cost of, of testing, especially like we've talked about the cost of LLM. But like there is, uh, cost considerations and also rep- repeatability-

Maude Lemaire 46:02 Yeah

Gergely Orosz 46:03 ... how lightweight it is, how much easier it is to control it and, and to, to even tweak it or, or-

Maude Lemaire 46:08 Yes. Yes, yes

Gergely Orosz 46:09 ... it sounds like to your solution, you could, you can later, if, if you find a different failing loads case, you can just rewrite it to, to that. It's not a big deal.

Maude Lemaire 46:15 Yeah. Yeah. It was really lightweight. Um, it just did what it needed to do. We created a, a pretty-- I mean, we evolved the system whereby you could create like routines to send to these like fake users to run through over time. At first it was just like a very sort of silly like JSON blob of like, "Call this thing, then call this thing, then call this thing."

Gergely Orosz 46:36 Yeah.

Maude Lemaire 46:36 I wrote a like mini recorder tool where I could, via slash command in Slack, say like, /start recording.

Gergely Orosz 46:43 Nice.

Maude Lemaire 46:44 Go through a whole flow in Slack. It would record every single API request I made across both to the-

Gergely Orosz 46:50 Mm-hmm

Maude Lemaire 46:50 ... um, normal web app backend and to our edge cache. It would request, it would record every single outgoing and incoming WebSocket message, and then I would sort of sort through it all to just the pieces that I wanted to simulate and then inject that into another tool that I wrote that like turned that into a script that I could then send to the simulated clients. Because ha- the other half of the battle was figuring out what are all of the APIs that get called when I do this one product flow, which is like not a small number. [laughs] And you wanted to be sure that you captured the entire thing, so that, that's where the, the recorder tool came in, came in pretty handy, and we used that over and over and over again. And I would, again, to go back to the product thing because I'm gonna keep hammering the product thing, I think like the best product managers are mindful of, like performance and reliability are important aspects of any given feature, and the best PMs know that and care about that and optimize for that. And I was trying, the last like several years that I was there, I was always trying to push more tools into the hands of PMs and say like, "Hey, you wanna add your use case to be automatically tested with QoE Pond with our load testing system? Here's the recorder tool. Go record the flow that you want us to test. Send us back the file. We'll plop it into the system, and we'll run it at the frequency that you want it to run at." Because they knew, you know, oh, files do this weird thing when we like pop open a canvas and it like does this thing and like, I don't know. I don't use canvases the way that like most of our users probably use them. You do. You're the PM.

Gergely Orosz 48:26 Mm-hmm.

Maude Lemaire 48:26 So like go record it and then bring it back to us, and then we'll just shove it into the system we wrote.

Gergely Orosz 48:31 Speaking of reliability, uh, after Slack, you joined GitHub as a principal software engineer, and this was 2024, so this

Gergely Orosz 48:38 was at a time where GitHub Copilot was already, uh, very big. Um, in fact, uh, Cursor and other competitors were, were already, uh, as, as tools were, were starting to get up there. The load on GitHub service has already started to increase-

Maude Lemaire 48:51 Yes

Gergely Orosz 48:51 ... and GitHub's reliability started to visibly become shakier, not as bad as it is, uh, these days. What did you observe inside of GitHub? Can you tell us like kind of the, the vibe wh- when you joined? You know, like if you can give us a bit of a, an understanding and appreciation of, of what GitHub has been built on, this, this massive Ruby monolith, and when you joined and what was it like?

Maude Lemaire 49:11 It was a really interesting time, uh, particularly because, um- One of my friends from Rent the Runway, actually, like, we're going, we're going way back, um, had been at GitHub for many years at that point. And we'd sort of been keeping tabs on each other's careers, and she told me, "Maude, we are going to run out of space." Like, GitHub is still primarily on data centers, and it's, like, increasingly less so over time.

Gergely Orosz 49:37 They, they, they used to have their own data centers.

Maude Lemaire 49:39 Yes.

Gergely Orosz 49:39 Now they're moving to Azure. This is public information-

Maude Lemaire 49:42 Yes

Gergely Orosz 49:42 ... they shared.

Maude Lemaire 49:43 Yes, yes, yes. Yes. Um, yeah, the move to Azure is, um, front and center, but still a significant... Or at least at the time, um, when we were talking, a still significant amount of load was going through the data centers that, like, um, they had whole teams managing the metal. And she was like, "We're, we're gonna run out of space. The space is running out, and we either... Well, we both have to figure out how to move to cloud, and we also have to figure out how to do less, um, because the, the, we're doing a bunch of dumb stuff, as everyone does-"

Gergely Orosz 50:15 Yeah

Maude Lemaire 50:15 ... "that we, like, shouldn't be doing, and we can make the system a lot more efficient with what we have to give, buy us more runway." Uh, I was like, "Ooh, this sounds like a very interesting problem. Would like to work on this problem." But I was also seven months pregnant at the time. [laughs] And so what that ended up being was I, I showed up, I was seven months pregnant, which is actually kind of, if you can swing it, a really great time, way to join a company because you are... You sit there and you're like, "Cool, I will just drop a bunch of opinions," like, you know, nothing very strongly held, but, "I will just drop a bunch of opinions on people. I will own nothing for the next two months, and then I'm going to go have a baby, and then I'll come back." [laughs]

Gergely Orosz 50:57 And see what happened.

Maude Lemaire 50:58 And see what happened. [laughs] And, uh, and in my first couple weeks, I was sort of floating around some spaces, and I, having done a bunch of database migrations at Slack and some of the stuff that we didn't talk about was w- but was moving from this, like, team-sharded, tenant-based model to a more dynamically sharded model with Vitess, where it turns out, uh, no, a team was too big.

Gergely Orosz 51:23 So, so v- you, you mentioned Vitess. What is Vitess?

Maude Lemaire 51:26 Vitess is a, it's essentially a wrapper on top of MySQL that lets you shard, um, more dynamically. And so you have a-

Gergely Orosz 51:33 So in- instead of just having, like, MySQL run on one database, uh, and when you outgrow it, good luck.

Maude Lemaire 51:40 [laughs]

Gergely Orosz 51:40 You, you, you manually move the tables-

Maude Lemaire 51:42 Manually, yes

Gergely Orosz 51:42 ... or copy them or whatever. It's a nightmare. It allows you, as it grows and as it outgrows, it will, I don't know, predict or it, it, it will now have a way to more seamlessly move over to multiple machines and-

Maude Lemaire 51:53 You can reshard, redistribute your data, uh, a lot more seamlessly-

Gergely Orosz 51:58 Got it

Maude Lemaire 51:58 ... than, than you would be able to otherwise.

Gergely Orosz 52:00 Yeah.

Maude Lemaire 52:01 Um, and you can choose different sharding keys more easily as well. And so, like, before, at Slack, everything was sharded by team. There's still data that is sharded by team, where team is the logical unit that makes the sense for, makes the most sense for fetching that data.

Gergely Orosz 52:13 Yeah.

Maude Lemaire 52:14 But actually turns out that the smallest, most logical unit was channel. And so we ended up resharding a bunch of our data to be, um, fetchable by channel ID rather than by team ID. And then that allows for much better horizontal sharding structure as well. Um, and the idea being that there is no channel that will ever be big enough, maybe one day it will be, [laughs] the bots keep posting in the channels maybe, um, that will have enough data associated with it that it can never run on a single-

Gergely Orosz 52:45 Shard, yeah

Maude Lemaire 52:45 ... ultra-large database.

Gergely Orosz 52:46 Yeah.

Maude Lemaire 52:47 Right? Um, it's a much better unit from a scaling perspective. So Vitess has, had some of its natens- nascency, uh, embedded in GitHub, actually. And it was really interesting because when I was interviewing at GitHub, uh, I spoke with a bunch of the principal engineers that I would later end up working with, and they were sort of trying to figure out, "Okay, we have Vitess, but I don't think we are actually using it to its potential, to where we should be using it." And so they were asking me a bunch of questions about, like, our Vitess setup and what we were doing. Caveating this with I did not ever operate Vitess, um, or deal with the internals. I was more of a, I'm a product platform person. And so I just, like, find that, like, no, we should not be sharding this thing by this thing anymore. Let's migrate it. And so that was my perspective on it, which was also mostly their perspective on it as well. And I came in and I started poking at the Vitess setup, and I was like, "What are we doing? [laughs] What are we doing here?" I was used to thousands of shards per, um, per sharding key. So say four, four channels, we have a bajillion, um, because there's a bajillion channels and you want to sort of spread load evenly-

Gergely Orosz 53:57 Yeah

Maude Lemaire 53:57 ... throughout those. And when I arrived, I was looking at the Vitess setup and there was, I mean, I don't know if this is still the case, but there were 16 shards for all of pull requests and issues.

Gergely Orosz 54:11 Across all of GitHub.

Maude Lemaire 54:12 Across all of GitHub. Only 16, which is not very many given the amount of data that we know is associated with that whole space. And my first question was like, "Wait, why do we only have 16 shards? [laughs] Oh my gosh, this makes me deeply, deeply nervous." And part of the answer was, we can't have more because we already have so many scatter queries. So those are queries that are coming in-

Gergely Orosz 54:40 What are square, sure-

Maude Lemaire 54:40 Yeah

Gergely Orosz 54:40 ... what are scatter queries?

Maude Lemaire 54:41 So those are queries that are coming in where you do not have the primary key in them. So say you are querying for a pull request or an issue, and you don't have the pull request ID, which is how this data is being, um, distributed. Well, you have to find it, so you have to go through, so you have to find and, and match the criteria. So say I'm querying for, um, all pull requests authored by me.

Gergely Orosz 55:07 Yeah.

Maude Lemaire 55:08 I don't know that that's an actual scatter, but this could be-

Gergely Orosz 55:10 Yeah

Maude Lemaire 55:10 ... in the scenario.

Gergely Orosz 55:11 Let's just assume.

Maude Lemaire 55:12 Let's just assume. Um, and so because I don't have a pull request ID, well, I'm going to hit every single one of those 16 shards.

Gergely Orosz 55:19 Wow, this is-

Maude Lemaire 55:19 [laughs]

Gergely Orosz 55:20 This will be bad on so many levels.

Maude Lemaire 55:23 Uh-huh. Because, well, so many people are doing it also

Gergely Orosz 55:25 And I, I, and I understand then why you don't want to have many shards, because if you have more shards, you would need to hit all of them. We're talking like... It's almost like a linear scan.

Maude Lemaire 55:33 Yeah. Yeah. Oh, no, it's, it's, it's quite, quite, quite inefficient. And again, I don't know that this is an actual example of one of the-

Gergely Orosz 55:40 This was hypothetical

Maude Lemaire 55:40 ... This is completely hypothetical, but there were still very many, um, significant scatter queries that were hitting almost every single shard before it knew that it could complete the query and return the result. Um, which, yes, I understand why that was a constraint then. You don't want to add [laughs] more shards. But then again, you also cannot make the shards much bigger, and they're under very heavy load. Uh, and so I was ready to, like, apply the playbook that I'd been using for the last several years of, like, well, let's just burn out, burn down the scatters. Let's just get-

Gergely Orosz 56:11 Yeah

Maude Lemaire 56:11 ... rid of the scatter queries.

Gergely Orosz 56:12 Don't have scatter queries.

Maude Lemaire 56:13 Just don't have scatter queries. That can't be that hard. Well, um, like, turns out that can be very hard, um, when you're dealing with a Rails monolith-

Gergely Orosz 56:22 Yep

Maude Lemaire 56:22 ... and you've got Active Record. And I always sort of, um, thought it was funny the number of... There were a lot of, uh, ex-Twitter folks at Slack early on, and everyone was like, "Ha ha, Active Record." And I was like, "What are you..." Like, before they did the whole-

Gergely Orosz 56:37 Yeah

Maude Lemaire 56:37 ... like, Scala Java thing, right? And I was like, "Okay, I guess, like, we're all, you know, joking about Active Record." I did not understand [laughs] until I, like, came face to face with it, and I was like, "Oh, I really don't like this. This is really terrible."

Gergely Orosz 56:50 So Active Record is, is Rails'

Gergely Orosz 56:52 abstraction layer for the database, right?

Maude Lemaire 56:54 Yes.

Gergely Orosz 56:54 So you define things in, I guess, code, and it kind of maps nicely to database, except it might be harder to fine-tune your queries, et cetera.

Maude Lemaire 57:04 Yeah, I mean, it, it does a very good job at hiding the underlying data model from you, which, again, pros and cons. I came from a world where, like, SQL was, you know-

Gergely Orosz 57:17 Yeah

Maude Lemaire 57:17 ... religiously written out by hand for performance reasons. And then I was being thrown into this world where at one point we were trying to figure out... We were doing a bunch of work around domain extraction. And so how do we rip out everything related to, say, pull requests from the Rails monolith and extract that into a separate Go subsystem, take everything that deals with pull requests, just, like, take... Make the requests go through, say, the monolith still, but then just, like, immediately send them over to the other system. Really standard sort of migration procedure. We started doing that. We started with stars, and I think that project was eventually abandoned. But I was working, I was pairing with someone who had, like, a ton of Rails experience. And I was like, "Okay, you gotta, like..." Because I, as someone who was so used to reading just code that was very sort of straightforward and didn't have so many levels of, of abstraction, I... It's a little bit daunting when you're going into a world where now there's abstractions upon abstractions upon abstractions. And I was working with her to try to rip out more of the stars-related queries inside of the Rails monolith. We'd found sort of the, the root caller that created this query that would call into the stars table. And it was, like, a five-line function, and I was looking at it with her on Zoom. And I said, "Wait, Kate, where, where is the call? Like, where, where is the SQL being made? Like, what is happening? It's a five-line function." And she's like, "Maude, that's the million-dollar question." And I was like, "What are we looking [laughs] at? What is happening?" Because I, I half expected to see... I mean, at this point, I think so many people use Prisma, right? And Prisma, you can sort of... It's not, you know, plain SQL written out as a string between quotation marks, but you can sort of see the structure of the SQL query being made. You do not get any of that in this case.

Gergely Orosz 59:15 Yep.

Maude Lemaire 59:15 It's just, it just looks like a regular function call. And in that moment is when I realized this is why this has been impossible to date, and there is... Because no one had... And why the problem has gotten so bad, because no one sees the impact of just writing a new line of code.

Gergely Orosz 59:35 And, and also fixing it now m- will probably... It's not as easy as just tweaking your abstraction layer.

Maude Lemaire 59:43 No.

Gergely Orosz 59:43 You would probably have to change how code is being written, being called, starting to call more either functions that call SQL, et cetera. Like, it's, it's just you took a technology that worked great when you were small. You were able to scale it as long as y- And, and they figured out how to s- how to grow it as long as things are growing linearly.

Maude Lemaire 1:00:01 Yeah, and I think one of the things that, like, blew my mind the most that, like, both gave me a lot of respect for how they've managed to date, um, and also filled me with much dread, was, uh, when you work inside of GitHub, obviously you have, like, the admin view of GitHub. And there's this bar at the top where you can load up a page on GitHub, and it'll tell you exactly which MySQL queries were run to populate the page-

Gergely Orosz 1:00:29 Mm-hmm

Maude Lemaire 1:00:29 ... which clusters they hit, um, how fast they all were-

Gergely Orosz 1:00:33 Mm-hmm

Maude Lemaire 1:00:33 ... totals around all of this stuff.

Gergely Orosz 1:00:34 It's like the internal employee view. Yeah.

Maude Lemaire 1:00:35 Yeah. Yeah, yeah, yeah. And the, what's called the files disambiguate page, which is sort of like the repo homepage that you land on with the-

Gergely Orosz 1:00:42 Yeah

Maude Lemaire 1:00:42 ... README. That page last, last I checked, you know, before, before I left-

Gergely Orosz 1:00:47 Before you left, like a year ago

Maude Lemaire 1:00:48 ... Actually less than that, but yeah. Yeah, yeah, yeah. So some, some amount of time ago, um, was, it was 100, over 100 MySQL queries to populate that page across at least three different MySQL clusters. And the thing with Active Record is that it only caches the output automatically if the query is exactly-

Gergely Orosz 1:01:11 Mm-hmm

Maude Lemaire 1:01:11 ... the same. And so if, say, you're fetching user information, like full, like blanket, like select star from users where user ID equals blank, and you have that full user. But then later on you do select star from users where user ID equals blank and something else, it's not gonna reach back into the results that it cached for that first query because they are not exact same queries. Every single count- Or at least when I last left. Almost every single count on, you see on a GitHub page is live calculated. None of those are cached. There's a lot of reasons why that is. [laughs] Mostly product reasons, actually. Um, but I... That's one of the first things that I was like, "But why?" And to go back to, like, my sort of silly emoji example of, like, the 10,000 and 1th emoji, no one cares. Why does anyone care? Right? And if you have low counts of things, sure, okay, maybe it matters that there's three open pull requests on, like, this very small repo that, like, no one really does anything with. But once it gets to, like, 99, uh, don't, just don't show anything. Just say 99 plus. Don't bother counting it. Why are we counting this?

Gergely Orosz 1:02:19 Really interesting that so far we talked about you, you did interesting thing at Slack, at, at GitHub. You also worked at Copilot

Gergely Orosz 1:02:26 code review.

Maude Lemaire 1:02:26 Yes.

Gergely Orosz 1:02:27 Afterwards, you worked at, at Cursor. Again, uh, they, they really pushed how we're coding with, with AI tools. But so far we've not really talked about AI. It just didn't-

Maude Lemaire 1:02:38 Right, yeah

Gergely Orosz 1:02:38 ... come up naturally.

Maude Lemaire 1:02:39 [laughs]

Gergely Orosz 1:02:39 And I wanted to ask firstly, what do you think AI so far... Uh, you have, you obviously use these tools all the time, uh, increasingly intensively. What parts of the job does AI not help you with?

Maude Lemaire 1:02:54 Mm. Mm-hmm.

Gergely Orosz 1:02:54 And what parts of it has it helped and has it changed how you work?

Maude Lemaire 1:02:58 Oh, gosh. Yeah. Um, it has not helped me with performance optimizations basically at all. [laughs] Um, at least the investigation part. Maybe the implementation part is a little bit faster.

Gergely Orosz 1:03:11 What once you know what you wanna do.

Maude Lemaire 1:03:12 But yeah, figuring out what I want to do. So say, um, I was looking at improving time to first token, um, time to first visible token.

Gergely Orosz 1:03:20 At Cursor.

Maude Lemaire 1:03:20 At Cursor, um, a couple of months ago where, um... And there's a bunch of factors that feed into how fast you start getting those tokens back. There's a significant amount of time that goes into when you are typing your prompt, um, and you hit submit. There's a significant amount of time that goes into building all of the context associated with the request that you are about to make to send to the LLM. Um, and then there's some amount of data that we are able to cache, some amount that we are not able to cache either for product reasons, privacy reasons-

Gergely Orosz 1:03:55 Mm-hmm

Maude Lemaire 1:03:55 ... but also strictly because of load and, like, again, cost of these systems. It is not cheap to say cache absolutely everything that you have ever queried Cursor for in Redis for 24 hours. Um, that costs a lot of money. So we're not gonna do that. Uh, and then eventually sending that request over to, uh, inference proxy, which then sends the request over to the underlying LLMs and, and providers. Given that there's so much that, um, happens, but also back and forth, a lot of this process is not really as synchronous as the experience leads you to believe. And in the regular flow, if you're lucky enough to be able to use HTTP2, which some poor folks out there are stuck on HTTP1, you establish that, um, gRPC streaming connection with the backend servers, and you're able to split that connection into, like, four or more concurrent streams and send and receive different bits of information across those streams. It's very complicated, and also AIs think that they're really good at instrumenting, um, these systems. Uh, they are not, uh, turns out. They think they're... It's like writing tests. They think they're really good at writing unit tests, but they're really good [laughs] at just confirming what they just wrote.

Gergely Orosz 1:05:14 Yeah.

Maude Lemaire 1:05:14 Which is why some of the skills that I ended up crafting run through, like, intensive series of mutation tests on the code to then force the LLM to realize, "Oh, I didn't actually test any of the edge cases. Let me rewrite those tests." [laughs] Thank you. So the LLMs right now I find are, uh, good at doing the really basic level in, of instrumentation, but not great at figuring out, especially in an asynchronous context, what makes sense to instrument how. I love traces. I think they're amazing. You can get so much really good debugging information from them, particularly from a perfor- performance perspective. We had this problem at Cursor where we had, like, too many and too little traces, and by that I mean that we had traces with millions upon millions of spans, which are completely useless. You cannot do anything meaningful with a trace that has that many spans. And then we also did the, um, very classic mistake that basically everyone does, which is we started sampling for cost reasons, but not in one place, which is again, really easy to have an LLM do because it's like, "Oh, like, find me a place where we can send fewer traces to save money." And it's like, "Cool, I'll just, like, sample it right before I send it to Datadog or whatever." No, no, no. Don't, don't, don't do that. Please don't do that. Always do it in the same place. The problem with duplication is just, like, so much worse when you're dealing with this. Um, and so one of the first things that I ended up having to do was, like, honestly manually instrument a lot of these flows, um, and actually read through the code myself and understand, "Okay, this chunk of, uh, processing is synchronous. This chunk of processing is asynchronous," and put all of those together. And one of the ways that I started doing that was I had it generate a, like, big sequence diagram of everything that goes into time to first token.

Gergely Orosz 1:07:19 Mm-hmm.

Maude Lemaire 1:07:19 So, like, from the time you hit submit to when you receive that first visible token, and it was huge. And I went to Staples, and I had them printed out on, like, big blueprint paper because I'm still a really analog person. And then from that I went through and I was reading the code and I was annotating in- On my big piece of paper, like-

Gergely Orosz 1:07:39 Mm-hmm

Maude Lemaire 1:07:39 ... okay, this, this is correct or this is incorrect. This piece we could probably do something about. This other piece, no, actually definitely needs to happen before this other piece. One of the other things that I was finding that it was doing, um, really painfully wrong is that it was finding sort of the classic things that could-

Gergely Orosz 1:07:56 What, what, what was doing? What the LLM was doing wrong?

Maude Lemaire 1:07:58 Oh, yeah, so what it was doing wrong was, um, finding... Well, wrong, wrong is kind of the wrong word. Um-

Gergely Orosz 1:08:05 No, let's, let's be opinionated.

Maude Lemaire 1:08:06 [laughs]

Gergely Orosz 1:08:06 I like it.

Maude Lemaire 1:08:07 Wait, it, naively, naively what it was doing-

Gergely Orosz 1:08:10 Yeah

Maude Lemaire 1:08:10 ... was say you have a bunch of things. You're finding a bunch of tasks that don't depend on each other, right? But they're latency sensitive. What's the classic solution?

Gergely Orosz 1:08:21 You parallelize them.

Maude Lemaire 1:08:21 You parallelize them. Um, however, again, product hat, right? What if basically no one is ever doing all four of those things?

Gergely Orosz 1:08:31 Then why are you doing all of them at the same time? Like, you're wasting resources, latency. You're waiting... You're, you're increasing load for no reason.

Maude Lemaire 1:08:39 And so I [laughs] I remember, I think it was, I for- I think it was Opus that was, like, very excited that it, like, found this opportunity to parallelize a bunch of these tasks, and I was like, oh, like, on first glance, sure, that seems like a good thing to do. Except that the problem, even if you parallelize them, the longest one is always gonna be the bottleneck regardless. And the longest one was always so much longer than the other three, and the other three tasks were being run such a small... 1% of users were doing any of the other three, and so it ended up being a complete waste of time. [laughs] Like, this was not the optimization that it was, like, selling to me as being, like, the panacea to improving time to first token. And a lot of the things that were, uh, actually quite helpful were really silly things like, I mean, silly, like increasing the TTL on what we stored in Redis where we could within reason.

Gergely Orosz 1:09:38 How do you use LLMs? Because so far, my sense is, talking through your career, you're, you look at a problem, you understand it, you figure out how to do it. You fail sometimes or make mistakes. You learn from it. You learn from others, including right now in the last few years where AI is increasingly everywhere. But as you're making systems more reliable, more performant, it sounds like you're still doing so much thinking, so much for... You're now, it, I sense that you're using the LLM on the side to generate some ideas, to say, "No, this is wrong. No, hmm, that was a good idea. Okay, let me use it." But unlike a lot of software engineers I talk to, es- even on this podcast-

Maude Lemaire 1:10:16 Yeah, yeah, yeah

Gergely Orosz 1:10:17 ... most of them are saying, like, "Oh, so much of my work that I used to do, the LLM does it. I now can do more things. I can focus on higher." I'm not getting that sense from you.

Maude Lemaire 1:10:25 Yeah, I mean, I'm spending all day, like, chatting with an LLM in some way, shape, or form. I think, like, I still do a lot of the normal engineering stuff, which is like, I'm on call. Sometimes I just have to fix a silly bug. Um, but I do tend to, I do a lot of sort of preliminary e- exploration, um, but then verification manually.

Gergely Orosz 1:10:50 And I, I sense that a lot. Like, it sounds like even when you get something from an LLM or whatever, you will verify, you will make sure that you understand that this is right. It sounds like you know how judgment of it's gonna be right or wrong, but you need to verify it, right?

Maude Lemaire 1:11:02 Yeah. Yeah. Because I think a lot of the problems that I work on are more of the, like, product platform-y problems.

Gergely Orosz 1:11:08 Mm-hmm.

Maude Lemaire 1:11:08 And the whole point is to solve a problem holistically and to create that foundation that you can build on top of, right? And if you only build the foundation to cover, like, 80% of the house, like, what's the point? [laughs] Because that other 20% that was not built on the foundation is going to be really shaky and impact the rest of the house that you did build on the foundation. And if you already did that over here, well, like, nothing's really stopping you from, say, like, tacking on another 10% off past the foundation and then that spiraling, right? And so I think especially, especially now that, like, I think AI's so good at just duplicating things in places because that seems fine and it doesn't have a sense of... It's not great at going, you know, "But why? But why?" But, like-

Gergely Orosz 1:12:04 Mm-hmm

Maude Lemaire 1:12:04 ... 20 levels removed, it leans more towards, "I will just tack on... I'll just build an, build an extra bathroom past the foundation. That's fine," right? Like, "You asked me to build an extra bathroom. That's where I'm doing it," rather than questioning the underlying fundamental assumptions, which are, should one build a bathroom not on top of a foundation? And those are the questions that I'm asking myself all the time, and I think part of it is because I'm coming from a product platform perspective.

Gergely Orosz 1:12:36 I have a sense that the product platform work, the platform

Gergely Orosz 1:12:40 work reliably performance, it will only be more important because, again, we'll have systems that have so much load, so, so, so much test. What advice would you have for a software engineer, uh, who m- may or may not have much experience, but who is excited about the idea of performance reliability and would like to get closer to eventually work on, on platform teams? What are paths that you've seen people make it onto platform teams that you worked on? How did they transition?

Maude Lemaire 1:13:08 I'm such a strong advocate for more product folks, like, more interchange actually between product folks and product platform teams. I've seen it so many times where product platform teams are spun up out of product engineers who care a lot about the fundamentals, right? They care about reliability. They care about performance already. They're trying to improve that and inject that into the way that they're building product all the time. And then I think the best product platform folks that I've worked with get to a point where they've Seen a pattern enough times in the things that they are building as product engineers, and they're like, "I wanna dig in and solve this problem across the board. Like, this specific problem I understand really deeply from having implemented it two, three, four times."

Gergely Orosz 1:13:51 Mm-hmm.

Maude Lemaire 1:13:52 "I wanna go solve it once and for all-

Gergely Orosz 1:13:54 Mm-hmm

Maude Lemaire 1:13:54 ... and be done, and, like, have this thing that other people can also use," right? They get annoyed with the toil, and then they go and maybe embed with a product platform team, or they, like, spin one up with some other folks, and then they do that. And then a lot of times, those folks end up staying on product platform teams and continuing to really like the product platform work, but then they get further and further detached from what product development actually feels like. And I think that's where you see, uh, sort of the chasm that ends up being created between product platform teams and product teams, is just there's not enough cross-pollination between the teams. And so one of the ways that I've seen that chasm merge back together is having, like, rotation cycles between those teams, sort of always having someone on the product platform team, having a core set of folks who are, like, maintaining the main systems and platforms that they've built. But sort of always having someone on rotation from a product team on the product platform team, bringing, like, "Oh, well, I don't know what you all are doing here, but, like, on product we're just, like, YOLO shipping all this AI slop all the time. Like, what? You're reading your code? What?" Um [laughs]

Gergely Orosz 1:15:00 Let's happen. It's a thing. It's a thing. We-

Maude Lemaire 1:15:02 [laughs]

Gergely Orosz 1:15:03 On, on, on product code reviews are, are slowly dying.

Maude Lemaire 1:15:06 [laughs] And, like, completely different, right? But if you're not close to that, then you also don't know how to bolster your platform to be the best that it can be so that they can do that. I see my role in this, you know, phase of, of the world right now as a product platform person to be... Like, the guardrails need to be better than they have ever been before.

Gergely Orosz 1:15:28 Yeah.

Maude Lemaire 1:15:28 Because product people need, I mean, need is a strong word, but should ideally be able to slop out whatever the heck they want.

Gergely Orosz 1:15:38 And, and the platform should stop them or warn them or-

Maude Lemaire 1:15:43 Yes.

Gergely Orosz 1:15:43 So, so do I understand that your suggestion is, like, look, if, if you today are on a product team, you're building products, you're, you're shipping to customers, to users, obviously understand what the business cares about, builds, you know, try to build quality stuff. But also ask yourself, like, are you caring about performance and reliability? If not, first of all, why not? [laughs]

Maude Lemaire 1:16:02 [laughs] Fair enough.

Gergely Orosz 1:16:03 And, and, and second of all, as you start caring about this, and if you are working with a platform team, start working with them, hang out with them, see how you can... These problems that you're having, can I help build a solution for that? If you have the option, transfer onto this team. It's, it's pretty useful. And if you are on this team, try to have a way to work a little bit, like implement a feature with the team, understand why they're shipping so quickly and so seemingly carelessly, because this will... Both ways will just make... Like, it sim- seems like we need, need both, especially now that product teams are pushed to ship a lot faster, and they are shipping faster, by the way.

Maude Lemaire 1:16:38 Yeah. Yeah.

Gergely Orosz 1:16:38 And there's n- new tools. Some of them work, some of them don't. And, you know, this is how we're gonna be better engineers and how we will build better systems that allow us [laughs] to, like, ship reliable and quality experiences, which I'm, I'm all for. I, I hate seeing quality degradations on software.

Maude Lemaire 1:16:52 Oh, no. No, I know. And I mean, like-

Gergely Orosz 1:16:54 But, but, but we can stop it, or we c- we can actually build better s- software, hopefully.

Maude Lemaire 1:16:58 Yeah. Yeah, I think so. I think it's, um... it is really similar to, for folks who are familiar with, like, the em- embedded SRE model, like, there's a reason why that's really, really effective, and I think that there's, uh, real parallels with product platform as well. I kept talking to my team about, like, I mean, we always had to fight against spaghetti code, right? So easy. But now we're at this point where, like, the spaghetti has, like, a life of its own, and it, like, actively wants to spill out of the plate. Whereas before people were like, "Okay, like I gotta kinda keep it on the plate." No, no, no, the spaghetti, like, wants out, and we have to almost, like, red team our own situation.

Gergely Orosz 1:17:39 We do. Well, Maude, thank you so much for this conversation.

Maude Lemaire 1:17:42 Thank you so much for having me.

Gergely Orosz 1:17:43 It was nice to get an insider look of what scaling up looks like inside of products we all know and use, like Slack, GitHub, and Cursor. Until now, I'll admit, I never thought much about custom emojis causing a kind of scale problem. But Maude's example of the CEO sending out a everyone gets a Friday off message to 100,000 users was a good visualization. Hundreds of people add a custom emoji, and this causes an event storm of custom emoji binary sent all across the network and messages. Slack decided to fix the problem with a very simple solution: limit each message to 23 custom emoji reactions. I'm sure they could have done a more proper fix, but who will notice a 24th custom emoji not appearing anyway? It was also fascinating to hear GitHub's limit of 16 shards and why they couldn't add more. It was because of scatter queries, queries with no primary key that had to run on each and every shard before they could return. More shards would have slowed these queries down even more. The solution, of course, is to not have any queries without primary keys. But when you work with a legacy system with hundreds or thousands of queries without primary keys, rewriting those queries takes a non-trivial amount of time, not to mention that even as you're rewriting, you might have new queries added with no primary keys. Finally, it was interesting to hear how while Maude is clearly AI pilled, I mean, having worked at GitHub and Cursor, she still verifies pretty much everything by hand. Because when doing platform work, she says that you want to have a stable foundation. She also found that LLMs tend to write tests that just confirm the code written. Maude is now a fan of using mutation tests to have them cover edge cases. Do check out the show notes below for related The Pragmatic Engineer deep dives on Cursor, GitHub reliability engineering, and performance topics. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks, and see you in the next one.