By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
The Pragmatic EngineerScaling Slack and GitHub with Maude Lemaire
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Inside Slack and GitHub scaling: WebSockets, sharding, Rails, AI
- Maude Lemaire describes scaling challenges at Slack, including failure modes like thundering herds and client-side event floods driven by WebSocket fanout and downstream reads.
- She explains how Slack evolved load testing from API-only approaches and expensive headless-Chrome simulation to lightweight Go-based simulated clients capable of maintaining 500,000 WebSocket connections.
- She recounts a risky-but-careful Slack database-table migration strategy using feature flags, double writes, backfills, and shadow reads, including an incident where an internal rollout briefly blanked everyone’s sidebar.
- At GitHub, she observed a Rails monolith with heavy query behavior and a sharding setup where pull requests and issues lived on only 16 shards due to the burden of scatter queries.
- She outlines how she uses AI tools heavily but finds them weak for performance investigation and instrumentation decisions, emphasizing product-aware tradeoffs and rigorous manual verification.
IDEAS WORTH REMEMBERING
5 ideasRealistic load testing for Slack required exercising the WebSocket layer, not just API RPS.
At Slack, load tests that only hit APIs missed critical failure modes tied to real-time delivery and downstream reads; the team needed large numbers of persistent WebSocket connections to exercise the end-to-end path.
Thundering herds in Slack could be amplified by emoji fanout and follow-on reads.
She describes a common Slack failure pattern: a single high-visibility message triggers many users to click in, fetch history, and add emoji reactions, with each reaction fanning out to many connected clients and ballooning traffic.
Event floods were user-specific and often hit power users hardest.
She distinguishes “event floods” as a client-side overwhelm scenario, including cases where a single user receives duplicate events (e.g., the same message delivered multiple times across many logged-in workspaces) and the client becomes unusable.
High-fidelity load testing can be prohibitively expensive and unreliable at massive scale.
Slack’s initial high-fidelity approach—running Slack in headless Chrome at scale in Kubernetes—was costly and operationally fragile, leading to frequent pod crashes and limited usable test time.
A lower-fidelity, scriptable client simulator made 500,000-connection testing feasible and evolvable.
They replaced headless Chrome with many lightweight Go processes that maintained WebSocket connections and replayed representative API/WebSocket/edge-cache flows; Maude also built a recorder that captured real client interactions to turn into reusable routines.
WORDS WORTH SAVING
5 quotesIt's not because you're good at something that you like doing it, and I think I conflated the two for a really long time.
— Maude Lemaire
Oh yeah, people had completely blank sidebars for about half an hour.
— Maude Lemaire
Test in production.
— Maude Lemaire
It was 37 cents per, uh, per headless Chrome instance per five-hour tranche of load testing.
— Maude Lemaire
But now we're at this point where, like, the spaghetti has, like, a life of its own, and it, like, actively wants to spill out of the plate.
— Maude Lemaire
SPEAKERS
Maude Lemaire
guestPrincipal software engineer and author of "Refactoring at Scale," with engineering experience at Slack and GitHub.
Gergely Orosz
hostSoftware engineer and writer behind The Pragmatic Engineer, hosting interviews on software engineering and scaling systems.
High quality AI-generated summary created from speaker-labeled transcript.
