OpenAIWhy AI needs a new kind of supercomputer network — the OpenAI Podcast Ep. 18
CHAPTERS
- 0:00 – 0:41
Making supercomputers train faster: the networking problem OpenAI is solving
Andrew Mayne sets up the episode’s core challenge: getting massive GPU supercomputers to behave like one machine during AI training. Mark Handley and Greg Steinbrecher preview a networking breakthrough that improves efficiency, reliability, and scaling for frontier training runs.
- •AI training performance is increasingly limited by networking, not just GPUs
- •Goal: faster, more reliable training so improved models ship sooner
- •Episode focus: Multipath Reliable Connection (MRC) and why it matters
- 0:41 – 3:35
Greg Steinbrecher’s path: from physics and quantum computing to GPU cluster systems
Greg explains how an interest in complex systems led him from physics and a quantum computing PhD into practical data center networking. He describes moving from simulations to production software that makes GPUs communicate efficiently during training.
- •Shift from academic models to real data center workload realities
- •Early work: exploring optical switching ideas, then discovering headroom in conventional hardware
- •Transition from simulating clusters to building the software stack for GPU-to-GPU communication
- •Current role: maximizing GPU utilization, handling failures, avoiding network bottlenecks
- 3:35 – 5:06
Mark Handley’s background: internet standards to data center networking innovation
Mark describes decades of networking research, including work that influenced modern mobile networking. He contrasts the slow, consensus-driven world of global internet standards with the faster iteration possible inside data centers.
- •Earlier work helped enable real-time communication standards used in 4G/5G
- •Standards move slowly because the whole world must agree
- •Data centers allow faster experimentation: you only need agreement within a deployment
- •Motivation: apply research ideas to practical, high-scale infrastructure
- 5:06 – 8:50
Why AI training breaks “normal” network assumptions (synchrony, lockstep, worst-case)
The conversation shifts to how AI training stresses networks differently than typical internet traffic. Training runs are tightly synchronized, so the slowest GPU or the worst network link can stall everyone, turning tail latency into the primary metric.
- •Internet workloads average out via many independent flows; training is one giant coordinated job
- •GPU communication is part of the computation, not a side effect
- •If one GPU slows or fails, all others wait—wasted time and cost
- •Optimization target becomes the absolute worst case (P100), not the average
- 8:50 – 12:04
Bottlenecks at scale: path collisions, hotspots, and a building full of failures
Mark and Greg detail why large-scale GPU networks become fragile: there are thousands of potential paths, and random selection can create severe hotspots. At the same time, component counts explode—millions of optical links—so failures become continuous and must be treated as normal.
- •Hierarchies of switching create thousands of possible paths between GPUs
- •Random path selection can create heavy contention and dramatic slowdowns
- •Scale guarantees frequent failures (links, switches, optics) even with top-tier gear
- •More network components than GPUs means failure rates rise quickly with cluster size
- 12:04 – 15:26
Where things ‘break’: mean time between failures collapses with bigger clusters
They quantify the reliability cliff: doubling system size roughly halves time between failures, and the network multiplies component exposure per GPU. The need for full bandwidth prevents simplifying the network, so engineering must assume frequent breakage while keeping GPUs busy.
- •Failure frequency rises as infrastructure scales (independent-failure assumption)
- •A single GPU implies many dependent network components (transceivers, lasers, multiple switch layers)
- •You can’t ‘taper down’ the network without starving GPUs and slowing training
- •Downtime directly delays model progress and wastes power/money
- 15:26 – 17:01
How MRC works (part 1): spraying traffic across many paths + packet trimming
Mark introduces the core MRC insight: distribute packets over many paths to equalize load and avoid hotspots. To handle reordering and loss ambiguity, MRC uses packet trimming—forwarding a tiny header when queues overflow—so the receiver can request immediate retransmission without waiting.
- •Multipath packet spraying achieves strong load balancing across the fabric
- •Downside of multipath: packet reordering makes loss detection ambiguous
- •Packet trimming replaces drops: payload is removed but header continues to destination
- •Receiver quickly detects missing payload and requests retransmission, reducing stalls
- 17:01 – 18:32
What users feel: faster research cycles, fewer job failures, more predictable performance
Greg connects MRC to user outcomes: training runs become faster and more stable, and researchers no longer need to think about networking quirks or placement effects. The overall release cadence improves because the entire training/deployment pipeline moves more reliably.
- •Better model quality and faster delivery through higher training efficiency
- •Reduced sensitivity to job placement (e.g., rack neighbors) and cluster variability
- •Fewer training interruptions; improved reliability during long runs
- •Accelerates OpenAI’s end-to-end research and deployment pipeline
- 18:32 – 21:44
How MRC handles failures: endpoints route around broken links in milliseconds
They explain the failure-resilience breakthrough: rather than waiting for routing protocols to reconverge, endpoints rapidly detect bad paths and stop using them. This avoids multi-second (or worse) convergence delays that would otherwise stall synchronized training.
- •Traditional routing reconvergence can take seconds to tens of seconds in the tail
- •MRC eliminates coordination requirements: endpoints independently avoid failed paths
- •Failure impact is spread thinly across many flows instead of catastrophically breaking transfers
- •Observed behavior: frequent link flaps during buildout became essentially invisible
- 21:44 – 22:41
Turning off routing protocols: static routing at massive scale to reduce complexity
Mark describes an operational simplification enabled by MRC: disable dynamic routing entirely and use static routing in switches. With MRC finding viable paths on its own, the network control plane becomes simpler and less failure-prone.
- •Dynamic routing adds complexity and new failure modes in switch software/control planes
- •MRC can infer which paths work without network-wide convergence
- •Decision: static routing tables at boot time; no ongoing control-plane updates
- •Result: simpler network management and fewer control-plane-induced incidents
- 22:41 – 25:09
Ecosystem and standardization: building MRC with partners across the stack
They outline the multi-company effort required to deploy MRC in real supercomputers, spanning data center builders and chip/network vendors. Reliability improvements translate directly into fewer “horror story” nights for on-call teams and less researcher disruption.
- •Collaboration with Microsoft (data centers) and vendors (Nvidia, Broadcom, AMD, Intel)
- •Standardizing the specification to ensure interoperability across hardware
- •Stability gains are visible both anecdotally (researcher feedback) and in failure stats
- •Reduced operational burden: failures still happen, but they’re masked effectively
- 25:09 – 27:23
Why OpenAI is making MRC an open standard (OCP): shared velocity, less fragmentation
Mark and Greg argue that open standards benefit everyone: OpenAI builds on Ethernet’s openness, and MRC should similarly become a shared layer for AI infrastructure. They also warn against a fractured supply chain where incompatible proprietary approaches slow industry progress.
- •MRC spec to be published via OCP as an open standard
- •Open standards accelerate the whole ecosystem and reduce duplicated effort
- •Avoiding fragmentation in AI infrastructure/supply chains benefits everyone
- •OpenAI still benefits as the industry improves the underlying tech faster
- 27:23 – 30:50
Scaling limits and design philosophy: ride Ethernet, push intelligence to the edge
They discuss how MRC can scale with Ethernet and what constraints remain—especially the speed of light. The core architectural move is relocating complexity from the network core to the endpoints, including source routing with IPv6 segment routing to keep switches “dumb” and reliable.
- •MRC builds on Ethernet; as Ethernet speeds evolve, MRC can evolve with it
- •Hard limits remain (latency and physics), requiring continuous engineering
- •Edge intelligence + simpler core improves reliability at scale
- •Source routing via IPv6 segment routing: packet header specifies the switch sequence
- 30:50 – 34:59
Efficiency gains: flatter networks, fewer switches, lower power, better work-per-watt
Greg highlights a less obvious benefit: multipath capabilities can enable simpler physical network designs with fewer layers of switches. This reduces cost and power draw, increasing the fraction of energy spent on GPU computation rather than networking overhead.
- •Multipath load balancing can allow smaller, flatter network topologies
- •Fewer devices and layers reduce power consumption and capex/opex
- •Improves useful work per watt by shifting power budget toward GPUs
- •Operational simplicity scales alongside performance improvements
- 34:59 – 37:38
Future demands and ‘compute in space’: latency, failures, and why Earth still wins for training
The episode closes by contrasting training vs inference and addressing the idea of putting AI compute in space. Mark and Greg emphasize that large-scale training is latency- and reliability-sensitive, requires frequent human maintenance, and is already hard enough on Earth—making space a tough sell today.
- •Training needs tight synchronization; space-scale latency undermines performance
- •High failure rates and limited repairability are major blockers for space deployments
- •Inference might be more plausible in remote settings than large-scale training
- •Pragmatic conclusion: keep building terrestrial compute to scale intelligence
