Networking giant Cisco was absent from the party that helped develop Multipath Reliable Connection (MRC), the OpenAI-developed transport protocol for large-scale AI training networks. But one of its engineers just built an emulated version using Docker containers instead of a data center.
MRC uses SRv6 to spray packets across network planes – the segment routing protocol developed back in 2016 by the Internet Engineering Task Force (IETF) in an effort led by Cisco. One of the vendor’s principal architects, Bruce McDougall, wanted to get hands-on with the new architecture and so, in his spare time, built an emulator to do just that.
McDougall’s software emulator tool simulates MRC over a segment routing over IPv6 (SRv6) dataplane to “give engineers and operators a feel for the multiplanar network design” and its packet spraying abilities.
McDougall wrote in a LinkedIn post that he didn’t have a hardware budget to get hands-on with real MRC gear. Instead, the simulator uses Containerlab to spin up dockerized software for open networking in the cloud (SONiC)-VS switch instances. SRv6-capable Alpine Linux containers act as hosts, arranged into a multiplane fabric to give the feel of an end-to-end MRC architecture.
The emulator can be accessed via GitHub under an Apache-2.0 license. MRC was developed by OpenAI alongside the likes of Nvidia, AMD, Intel, and Broadcom. The protocol is designed to speed up AI model training on vast clusters, reducing packet delays and congestion when connecting tens-of-thousands of graphics processing units (GPUs).
The concept was formerly unveiled in May, though it was inadvertently detailed in a Microsoft blog caught by SDxCentral last November. The hyperscaler was using the protocol to optimize packet paths across its mammoth Fairwater AI data center in Atlanta. Beyond that unintentional reveal, MRC has been deployed across Oracle Cloud Infrastructure’s supercomputer in Abilene, Texas, which OpenAI uses to train its frontier models.
In the weeks since its formal debut, Nokia lauded MRC in a blog post that labeled the protocol as “an important evolution in Ethernet transport.” Cisco has sought to publicly position itself in line with the protocol, with an official post dated mid-May outlining it as essentially being built on a foundation the vendor helped build.
The best of multiple worlds #
MRC is vendor-agnostic, so the protocol can be applied to hardware ranging from Nvidia’s Spectrum-X and Broadcom Tomahawk to its own.
In a recent conversation with SDxCentral, Broadcom’s VP of products Hasan Siraj said MRC allows for more dynamic routing in AI infrastructure.
“In a distributed cluster, routing was dynamic; people have used border gateway protocol (BGP), and your path selection is very hop-by-hop; every hop decides what the path should be moving forward," Siraj explained. “In the case of MRC, you basically introduce the concept of a controller. You're more centralized; instead of using BGP, you're using SRv6, which is a path-based selection, and your path management is done end-to-end; you're determining the path end-to-end up front.”
Instead of reinventing the wheel, Siraj – a Cisco alum – said that the folks who developed MRC took the best from a variety of industry offerings, including SRv6.
“It's not something that we have tried to invent completely from scratch," Siraj said. "We know that there is a lot of work that has been happening out in the industry [like] Ultra Ethernet Consortium … a lot of these techniques we have taken them and extended them and added SRv6 source-based routing to support very large fabrics.”