By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
The Pragmatic EngineerScaling Slack and GitHub with Maude Lemaire
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:00 – 3:18
What it costs to simulate internet-scale usage: 500,000 WebSocket connections and GitHub’s 16 shards
The episode opens with two punchy scaling anecdotes: load testing Slack at 500,000 concurrent WebSocket connections, and discovering GitHub had only 16 shards for all pull requests and issues. These examples set up the theme of scaling systems where architecture and operational constraints collide with real-world growth.
- •Slack needed to simulate 500,000 active WebSocket connections for load testing
- •An early approach spun up many headless Chrome instances and was extremely expensive per test run
- •GitHub’s pull requests and issues data was spread across only 16 shards across the whole service
- •The shard count was constrained by the presence of many scatter queries
- 3:18 – 7:29
Maude’s path into engineering: from microbiology plans to loving the power of code
Maude describes growing up around computers through her father, who taught computer science, while initially planning to pursue genetics research. A switch away from heavy memorization in biology led her to an intro computer science class—where she quickly realized she loved programming.
- •Maude grew up around computers and early exposure included computer-related materials at home
- •She started university intending to study microbiology/genetics and loaded up on demanding coursework
- •She realized being good at memorization didn’t mean she enjoyed it, prompting a pivot
- •Taking intro computer science flipped her mindset: typing code and making the computer “do things” felt empowering
- 7:29 – 9:16
First job at Rent the Runway: frontend work, women leaders, and early performance curiosity
Maude recounts returning to Rent the Runway after an internship, motivated by the company’s mission and the chance to learn in an environment with many women engineers and leaders. She also notes this period helped spark her interest in performance work, even if it wasn’t always part of her day job yet.
- •Rent the Runway combined programming work with an industry she was excited about
- •She was inspired by senior women leaders and wanted to learn from them
- •She mostly worked on frontend at the time and enjoyed the team environment
- •She began developing a strong interest in performance work during this period
- 9:16 – 13:58
Slack in 2016: early scaling limits and the push toward enterprise performance
The host frames Slack’s early constraints, including limits on very large channels and enterprise usage patterns. Maude describes arriving just before Slack Enterprise GA, with IBM already trialing it and pushing the system into new scaling territory that required a dedicated enterprise performance team.
- •Slack’s early architecture struggled with very large rooms/channels at the time
- •IBM’s growth in usage pressured Slack once Enterprise reached general availability
- •A dedicated enterprise performance team formed as the company confronted frequent instability for big customers
- •Maude joined performance work early and learned by being one of the most junior members on high-seniority teams
- 13:58 – 17:07
Slack’s early architecture: team-based sharding, dedicated chat servers, and Enterprise complexity
Maude explains Slack’s original model: data and chat infrastructure were associated with a workspace/team, and large customers could be placed on bigger databases or dedicated servers. Enterprise introduced cross-workspace needs (like DMs across teams), which created a complicated overlay and heavy load on shared enterprise-level data.
- •Slack’s data was sharded by team/workspace; large teams could be given larger databases
- •WebSocket connections were tied to servers associated with a company, concentrating load for busy organizations
- •Enterprise introduced an overarching entity to allow cross-team collaboration like DMs
- •Enterprise-level channels and DMs drove many users to query the enterprise shard simultaneously
- 17:07 – 18:44
The channel membership model: multiple tables, expensive joins, and why it hurt mentions and sidebar load
Slack treated DMs, public channels, private channels, and later group DMs as separate storage concepts, which made key product experiences expensive. Loading the sidebar and calculating mention badges required complex queries across multiple tables, becoming a performance and scalability pain point under enterprise usage.
- •Different conversation types lived in separate tables: DMs, public channels, private channels, and group DMs
- •Populating the sidebar required a union/join across these tables rather than a simple membership lookup
- •Mention/badge calculation was latency-sensitive and depended on complex cross-table queries
- •The data model reflected reasonable product/security intuitions but became costly at scale
- 18:44 – 26:29
Migrating live data at Slack: abstraction first, then feature flags, double writes, shadow reads—and one memorable mistake
Maude describes how she approached consolidating the channel membership access patterns across many call sites: first by creating a minimal function-based layer, then moving usage behind feature flags. She details a careful rollout strategy with double writes, backfills, and shadow reads, plus a production incident where Slack’s internal users briefly had blank sidebars.
- •Slack’s codebase had many embedded SQL queries and no ORM; migrations meant hunting many call sites
- •The first step was creating a minimal abstraction layer by copy-pasting query patterns into functions
- •Rollout relied on extensive feature flags, double writes, backfills, and shadow reads to compare old vs new results
- •A misstep during an internal rollout caused blank sidebars for about half an hour until the flag was flipped back
- 26:29 – 32:23
Load testing Slack realistically: thundering herds, WebSocket fanout, and product-driven priorities
The conversation shifts to load testing methodology: Maude emphasizes understanding real usage patterns and where systems historically fail. She explains Slack’s classic thundering herd scenario—large-scale reactions and channel reads triggered by a single announcement—and how Slack prioritized which WebSocket messages to keep under load.
- •Load testing starts with identifying real-world patterns most likely to break the system
- •A thundering herd in Slack can be triggered by a single announcement to a huge channel
- •Fanout makes reactions costly: each emoji reaction can multiply into massive WebSocket traffic
- •Slack prioritized message types over WebSocket under heavy load, allowing less important events (like emojis) to be dropped
- 32:23 – 38:29
Event floods vs thundering herds: when the client melts (and why power users suffered most)
Maude distinguishes event floods from thundering herds: event floods are client-specific overload from too many incoming WebSocket events. She shares an IBM admin example where being logged into many workspaces caused duplicated messages without deduplication, overwhelming the client and user experience.
- •Event floods are when an individual client is overwhelmed by inbound events over WebSocket
- •Power users were especially vulnerable, and they were also often Slack’s strongest advocates
- •An IBM admin who belonged to many workspaces could receive the same enterprise message many times
- •Lack of deduplication made the client experience degrade severely, including frequent client crashes
- 38:29 – 45:08
From headless Chrome to lightweight clients: building a practical 500,000-connection load test system
Maude explains why earlier load tests weren’t sufficient: they validated API and database throughput but didn’t include long-lived WebSocket connections. Slack’s initial high-fidelity approach used headless Chrome in Kubernetes pods, but it was fragile and extremely expensive; they replaced it with many small Go processes that simulated critical client behaviors at lower cost.
- •Earlier load testing focused on API/database load but missed WebSocket-driven failure modes
- •Simulating Slack usage realistically required 500,000 active WebSocket connections
- •Running Slack as headless Chrome in Kubernetes was costly and operationally painful due to resource demands and crashes
- •A lower-fidelity approach used many Go processes to establish WebSockets and perform key API flows while still exercising edge caches and backend APIs
- 45:08 – 48:31
Evolving the load testing tool: recording real workflows and involving PMs in performance coverage
The Slack load testing approach evolved into scripting reusable routines for simulated users. Maude built a recorder workflow using a slash command to capture API requests and WebSocket messages from real product flows, then curated them into scripts, aiming to make it easier for PMs to contribute realistic test scenarios.
- •Simulated clients ran scripted routines representing real product workflows over time
- •A recorder tool captured API calls to the backend and edge cache plus WebSocket traffic for a flow
- •Recorded traces were filtered down to key steps and turned into scripts for large-scale simulation
- •Maude wanted PMs to help define and contribute high-value workflows to be tested regularly
- 48:31 – 51:23
Joining GitHub in 2024: capacity pressure, datacenter constraints, and ‘doing less’ to buy runway
Maude describes joining GitHub during a period of rising load and visible reliability strain. She shares that capacity and efficiency were top concerns, with pressure from limited datacenter space and the need to both migrate toward cloud and reduce wasteful work in the system.
- •GitHub was under growing load pressure and reliability concerns were becoming more noticeable
- •A key internal worry was running out of physical datacenter space, increasing urgency
- •The move toward cloud was central, alongside efforts to make the existing system more efficient
- •Maude joined while pregnant and used the initial period to observe and share perspectives before taking leave
- 51:23 – 1:00:01
The Rails monolith scaling trap: Vitess, 16 shards, scatter queries, and why Active Record made it hard to fix
Maude compares Slack’s experience with heavy sharding to GitHub’s surprisingly small shard count for pull requests and issues. She explains scatter queries—queries that lack the sharding key and therefore must hit every shard—and why the Rails/Active Record abstraction made query behavior hard to see and even harder to systematically eliminate.
- •GitHub used Vitess but had only 16 shards for pull requests and issues, which felt risky given the data volume
- •Scatter queries happen when requests lack the sharding key and must fan out across all shards
- •The presence of many scatter queries limited the ability to increase shard count without worsening performance
- •Active Record’s abstraction hid the underlying SQL, making it difficult for engineers to see the cost of code changes and systematically remove problematic queries
- 1:00:01 – 1:02:19
Observability lessons at GitHub: seeing query explosions and questioning live counts
Maude describes GitHub’s internal tooling that shows which MySQL queries run for a page and where they go. She highlights how some pages triggered over 100 MySQL queries across multiple clusters, and notes that many counts shown on GitHub pages were live computed, which she questioned from a cost/performance perspective while acknowledging product tradeoffs.
- •GitHub’s internal admin view exposed query lists, clusters hit, and timing for page loads
- •A repository homepage-type page could trigger over 100 MySQL queries across multiple clusters
- •Active Record caching depended on exact query matches, limiting reuse when queries differ slightly
- •Many counts were calculated live, and Maude questioned why high counts needed exact computation given the cost
- 1:02:19 – 1:12:36
How Maude uses AI: helpful for implementation and drafting, weak for performance investigation and instrumentation judgment
Maude explains that AI tools haven’t helped her much with performance investigation, especially deciding what to instrument and how in asynchronous systems. She shares examples from work on time to first visible token where she had to manually reason about sync/async boundaries, manage trace volume, and avoid inconsistent sampling practices.
- •AI helped less with performance investigation than with implementation once a plan is known
- •Instrumenting asynchronous flows required careful human reasoning; naive AI-generated instrumentation was insufficient
- •Tracing challenges included unusable traces with too many spans and mistakes like sampling in inconsistent places
- •She used manual analysis, including printing large sequence diagrams and annotating them while reading code
- 1:12:36 – 1:19:28
Product/platform engineering paths: rotations, cross-pollination, and building guardrails for faster shipping
Maude advocates for stronger interchange between product engineers and product-platform teams to prevent a growing chasm. She argues rotations help platform teams stay grounded in real product development pressures, while product engineers who repeatedly hit the same reliability/performance problems are often best positioned to build durable platform solutions.
- •Many effective platform engineers start as product engineers who repeatedly encounter the same foundational issues
- •Platform teams risk drifting away from product realities without regular cross-pollination
- •Rotations can help: maintaining a core platform group while bringing in rotating product engineers
- •With faster shipping pressures, Maude sees platform guardrails as increasingly important to prevent runaway complexity
