Building resilient networks for AI supercomputers

By Valerie Cutts and Jithin Jose

Last fall we introduced Fairwater, the world’s most powerful AI datacenter. Delivering a system of this scale required rethinking how Azure designs supercomputers, especially the scale-out network. Today, we are sharing more about the networking innovations that have made Fairwater possible.

In this post, we share what’s unique about networking at extreme GPU scale and the system-level design choices required to enable large synchronous training jobs to run reliably, even during network failures. We are also publishing, in partnership with others, the open-source Multipath Reliable Connection (MRC) specification and software interfaces, and open-sourcing the associated libraries.

Fault-tolerant scale-out networking

At extreme scale, synchronous training amplifies the impact of routine network faults, turning packet drops, slow links, and partial failures into stalls, restarts, and wasted GPU time. As we describe in our MRC paper, the path forward is to treat failure as normal and design the network as an integrated system that:

Scales to 100K+ GPUs using a two level, multi-path topology to enable enough redundancy
Balances load evenly across the fabric to prevent congestion
Recovers predictably and gracefully during failures
Uses less power than three- or four-layer single-plane topologies

As outlined in the Multipath Reliable Connection Specification, Microsoft partnered with AMD, Broadcom, Intel, NVIDIA and OpenAI to jointly address this problem, focusing on changes to transport and network design needed to support training at extreme scale. Instead of relying on lossless fabrics and dynamic routing, we collectively designed, built, and deployed Multipath Reliable Connection (MRC), which draws upon lessons from the Ultra Ethernet Consortium (UEC), and paired it with a multiplane network topology, enabling reliable training jobs even when links, switches, or paths fail. The endpoint–driven transport created a simpler, more resilient network that delivers:

More resilient, predictable training at very large scale
Large training jobs continue making steady forward progress despite routine network faults, reducing stalls and restarts and improving time-to-train as cluster size increases.
Better utilization of expensive GPU infrastructure
By avoiding tail latency amplification and repeated recovery cycles, GPUs spend more time doing useful work instead of waiting on synchronization or replaying lost computation, improving overall efficiency and cost effectiveness.
Automatic adaptation at machine timescales
Failure detection, load balancing, and recovery happen fast enough to keep up with the rate and complexity of faults in 100K+ GPU systems, well beyond what manual intervention or control-plane convergence can achieve, allowing the system to remain stable as scale increases.

In the Fairwater supercomputer, enabling graceful degradation in the scale-out network improves training throughput versus traditional transports and architectures. In combination with a multi-plane topology design, MRC increases the time that installed NVIDIA GPUs perform useful computation.

A shift in philosophy: End-to-end control

The central design decision behind MRC is to shift responsibility for load balancing and failure handling from complex network switch control planes to the network endpoints with end-to-end controls. The network endpoint controls the path selection and can optimally use a set of paths based on feedback from the network.

MRC extends the RoCE Reliable Connection (RC) transport to support true multipath operation. Instead of binding a queue pair to a single path, MRC sprays packets across many paths simultaneously, making performance far less sensitive to any single slow or failed link. Several design elements are critical to enable end-to-end control.

Every packet carries enough information for the receiver to place it directly into memory, even if packets arrive out of order.
Selective acknowledgments enable rapid retransmission of only the packets that were lost.
Packet trimming signals network congestion swiftly without forcing full packet drops, enabling efficient congestion control.
MRC disables Priority Flow Control (PFC) entirely and runs Ethernet in best-effort mode, avoiding global pauses that can devastate tail latency or lead to fabric-wide deadlock behavior.
The system enables seamless self-recovery from network hardware failures.

The result is a transport protocol that expects loss, adapts quickly, and continues making progress even when parts of the fabric misbehave.

Rethinking topology: Multi‑plane design

Transport alone is not enough. To complement MRC, we implemented a two-tier, multiplane network topology in Fairwater using high-radix switches. Our network design splits each NIC into multiple lower speed ports (i.e. eight x 100 Gbps) and builds multiple parallel network planes. This multi-plane design enables a more compact topology, as opposed to a traditional three-tier Clos Network running at 800 Gbps/port. Our 2-layer multi-plane topology design offers several advantages:

Enables connecting 100K+ GPUs with just two tiers of network.
Lower latency since packets traverse fewer switches.
Reduced impact of network issues on overall job completion, while we see single switches connected to more servers, the individual impact is reduced by spreading the failure, decreasing the performance impact to the overall job.
Reduced hardware and power costs compared to designs with additional network layer, without compromising on GPU scale.
Most importantly, the network becomes more tolerant of partial failures, so jobs continue with slightly reduced bandwidth rather than failing outright.

Multiplane networks work efficiently only when traffic is evenly distributed across all planes and paths. This is where MRC’s packet spraying and path aware congestion response is essential.

Figure 1: Example of two-tier multi-plane topology using 512 x 100 GbE switches: 512 T0s x 256 NICs = 131,072 NICs

Static SRv6: Fewer moving parts, more predictability

In many data center networks, switches rely on Border Gateway Protocol (BGP) or other dynamic routing protocols. We removed dynamic routing from our design; instead, packets are source-routed using IPv6 Segment Routing (SRv6). Each packet encodes the end-to-end network path using compact microsegment IDs (uSIDs).

At first glance, static routing seems counterintuitive in failure-prone environments. At extreme scale, however, dynamic routing is more of a liability than an asset. Namely, if two or more switches try to reroute packets at the same time, network behavior becomes unpredictable and harder to diagnose. Interactions between adaptive routing and adaptive transport can be hard to resolve and harder to debug at larger scale.

MRC, on the other hand, handles path health and rapid failover at the transport layer. When static routing is used, it enables precise health feedback for each of the different network paths from that network endpoint. Because probe (test) packets follow the same paths as data packets, operators gain accurate, ground-truth insight into fabric health without depending on switch control planes, which are themselves a common source of failures.

Additionally, SRv6 routing allows network operators to utilize out-of-band monitoring frameworks to accurately identify link failures and device faults, which has been particularly valuable in managing large-scale AI clusters. Static SRv6 ensures paths are deterministic, making problems easier to reproduce, debug and, ultimately, more stable over time.

Failure as a normal operating condition

In production, failures are expected by design—link flaps, partial failures, and even switch reboots are routine at this scale. With MRC, many of these events no longer impact training workloads. Repair actions proceed in parallel, while MRC dynamically routes around failed paths. As repairs complete, MRC discovers and validates the restored paths before seamlessly reintegrating them—entirely transparent to the training application. In summary:

Systems degrade gracefully
Losing a NIC port reduces available bandwidth proportional to the lost port but does not crash jobs
Flapping T0–T1 links often go unnoticed by applications
Switches can be rebooted without coordinated drain or rerouting of the system

For massive scale training runs, this translates into higher effective uptime, fewer interrupted jobs, and more training throughput.

Figure 2: Bidirectional-bandwidth measurements with pt-pt RDMA Perftest while a T0 switch was taken down. Results indicate that the overall bandwidth dropped in proportion to the T0 switch bandwidth, but without failing the job

Figure 3: Bidirectional-bandwidth measurements with RDMA Perftest while T1 switch is failed and restored. Results indicate that no impact in performance as MRC was able to route around the bad switch

Measured results at scale

This is not just a thought exercise: Microsoft and OpenAI have both run extensive experiments and record-scale training jobs, showing only brief, bounded performance dips during significant network faults, followed by rapid recovery. Microbenchmarks demonstrate near line-rate bandwidth and predictable latency, even under injected loss. OpenAI describes their scale results in a recent blog post, consistent with what we observe. Taken together, multi-plane MRC with SRv6 delivers better load balancing with fewer queue pairs and substantially higher resilience to packet loss, enabling millions of networking links to connect hundreds of thousands of GPUs.

Figure 4: NCCL Send-Recv Benchmark Results with 42,020 GPUs each with 800 Gbps MRC NIC showing up to 92% of theoretical peak bandwidth for large message sizes

What this enables

Taken together, MRC, multiplane topologies, and static SRv6 form a coherent strategy for building AI supercomputer scale-out networks that keep large synchronous training jobs moving forward under real-world fault conditions. Instead of treating loss, link flaps and partial failures as events that trigger stalls or restarts, the system is designed to fail gracefully and reach 100K+ GPUs scale at high utilization. This design approach has been deployed in Fairwater and elsewhere to train state-of-the-art models, where the result is more predictable performance for large jobs with higher effective GPU utilization. The core takeaway is simple: by assuming failures will happen and, designing for them explicitly, events that would otherwise be catastrophic become minor, manageable perturbations.

Join us in advancing resilient AI infrastructure
To help the broader ecosystem adopt these capabilities, Microsoft is joining key partners in releasing the MRC specification to the Open Compute Project and open sourcing key components:

libMRC: MRC transport APIs
NCCL MRC plugin: enables NCCL to run over MRC transport
MRC shim library: enables compatible verbs applications to run over MRC with no code changes
MSCCL++ with MRC support: MSCCL++ library with MRC support
SONiC SRv6: enhance SRv6 with open NOS for high performance AI Ethernet

We encourage others to review these contributions to the public, share feedback, and ultimately adopt these capabilities within the broader ecosystem of AI networking products, infrastructures, and workloads.

Acknowledgements

Advancing AI at this scale requires collaboration across the industry. At Microsoft, we value our partnerships with AMD, Broadcom, Intel, NVIDIA, and OpenAI, and our shared commitment to continuing to evolve MRC alongside the broader community.

References:

This article was originally published by Microsoft's Azure High Performance Computing Blog. You can find the original article here.

Tags: Artificial Intelligence, Backup/Recovery, Cluster, Networking