I wrote [before](https://fergusfinn.com/blog/infiniband-roce-rdma/) about
‘scale-out’ fabrics — [InfiniBand](https://en.wikipedia.org/wiki/InfiniBand),
[RoCE](https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet), and more
generally RDMA — the networks and paradigms that stitch thousands of machines into a datacenter island.
This post is about the other kind. The names come from an old argument about servers: when you need more computer, you can either buy more machines and scale out, or buy a bigger machine and scale up. A scale-up fabric is an attempt to connect processors so tightly that they can behave as one. Inside a modern (NVIDIA) GPU machine the fabric is NVLinkThe terms are a bit frustrating really. Because you can scale up your LLM inference domain over your scale-out network. And you can certainly scale out inside a scale-up domain..
Fast, or far #
Let’s follow the physics of sending bytes over a link. At the lowest level: an NVLink is a bundle of wires, set up as [differential
pairs](https://en.wikipedia.org/wiki/Differential_signalling). Each pair is
driven by a [SerDes](https://en.wikipedia.org/wiki/SerDes): a circuit that
turns wide, slow on-chip data into one very fast signal on the wire.
The signal on the wire is just a voltage changing over time. To turn it back into zeros and ones the receiver has to sample it at the right instants. Before deciding what to put on the wire, we have to decide where the receiver’s sense of time is going to come from.
Length-matching meanders next to a BGA package
(source). One answer is to send a clock pulse, separate to the data, on its own wire, and sample as the clock ticks (a forwarded clock). This is how [DDR
memory](https://en.wikipedia.org/wiki/DDR_SDRAM) and
[HBM](https://en.wikipedia.org/wiki/High_Bandwidth_Memory) work. But it only
works if the clock’s ticks arrive close enough to all the signal bits for the receiver to sample them cleanly. The receiver can deskew small differences, but every data wire still needs about the same delay and, in practice, about the same length. This is why PCBs are covered in squiggles. At a hundred gigabits per second, one bit occupies about ten picoseconds: a millimetre or two of trace. That kind of matching is possible across centimetres, and gets much harder across metres.
We can instead synchronize each wire individually, sending its clock over each signal wire itself. We know that every transition in the signal (each 0-to-1 or
1-to-0) marks a bit boundary. So we can use a circuit at the receiver ([clock
and data recovery](https://en.wikipedia.org/wiki/Clock_recovery)) that keeps a
local oscillator locked to those transitions. The problem is that it works only if the transitions keep coming. If you send a million zeros in a row, the oscillator can start to drift. To keep the data transitioningStatistically, not provably. In principle you can craft data that XORs to a long run of zeros (“killer packets”, seen in SONET). Modern links use long scrambler polynomials to make this impractical., we XOR it with a pseudo-random sequence that both ends can generate
(scrambling). Top: each edge in the data re-centres the receiver’s clock, but across the run of zeros it free-runs (drawn a few percent fast) and the samples slide off-centre until one lands in the wrong bit. Bottom: the same payload scrambled, the edges never stop.
Once we can carry a timing signal, we need to put our 1s and 0s on the wire. One of the simplest ways is: high voltage for a 1, low for a 0. That scheme is called NRZNon-return-to-zero: return-to-zero being a version where you return to zero between every signalled bit.
To signal faster on a single wire under NRZ you have to tick faster. A trace on
a board works like a [low-pass
filter](https://en.wikipedia.org/wiki/Low-pass_filter): the copper, and the
insulation around it, both absorb more at higher frequenciesThe copper loses because high-frequency current crowds into a thin shell
at the conductor’s surface (the [skin
effect](https://en.wikipedia.org/wiki/Skin_effect)). The insulation loses
because the signal’s energy travels through it via the electromagnetic field, and the material’s molecules absorb some of it as the field alternates
([dielectric
loss](https://en.wikipedia.org/wiki/Dielectric_loss))., adding
attenuation that the receiver has to make back. It does this with filters that re-boost the high end, but this boosting amplifies the high-frequency noise along with the signal.
Rather than keep paying that rising attenuation, at some point it’s better to
try to send more bits per tick.
[PAM4](https://en.wikipedia.org/wiki/Pulse-amplitude_modulation) signals four
voltage levels instead of two, so each tick carries 2 bits of information. The cost is that the levels sit a third as far apart. As the level spacing gets closer, errors in decoding start to become more likely.
PAM4 links aim for roughly one bit in ten thousand wrong before correction, where NRZ links without correction can aim for better than one in a trillion. At these speeds an error rate of one in a trillion means an error every ten seconds or so, which is rare enough you can manage it at the link layer with checksums and resends. One in ten thousand means millions of errors a second, which is far too many to handle by resending. So PAM4 is usually paired with forward error correction. This costs latency, since the decoder has to collect a whole block before it can correct anything.
Instead of signalling furiously on a single wire, can’t you just add more wires? Yes, up to a point. Fast signals have to leave from the die’s edge: a SerDes buried mid-die would have to thread its traces out past all the stuff around it. Off-chip bandwidth therefore scales with the die’s perimeter (its shoreline) while compute scales with its area. The [reticle
limit](https://en.wikipedia.org/wiki/Photolithography) caps the die at
roughly 26 by 33 mm: about 120 mm of shoreline. The [HBM](https://en.wikipedia.org/wiki/High_Bandwidth_Memory)
interfaces, NVLink, PCIe, and C2C all compete for that finite edge.
The other variable is distance. Higher frequency signals travel less far: passive copper carries 100 Gb/s per pair about two metres, and 200 Gb/s about oneThese are IEEE spec objectives: IEEE 802.3ck specifies
100G-per-lane passive copper to at least 2 m; the follow-on
[P802.3dj](https://www.ieee802.org/3/dj/index.html) carries a 1 m objective at
200G per lane, and TE and Intel demoed 224G PAM4 over a 1 m passive cable at
[OFC 2023](https://opticalconnectionsnews.com/2023/03/ofc-2023-te-intel-demo-224g-pam4-over-passive-copper/).
Longer is possible with effort: [Marvell](https://www.marvell.com/blogs/224g-long-range-serdes-scale-up-scale-inside.html)
has driven 2.5 m of cumulative copper with a long-reach 224G SerDes.. To go further requires extra hardware steps. A redriver is an analog amplifier, and a retimer a kind of digital one: a full receiver and transmitter back-to-back decodes the bits and relaunches the signal anew.
The ledger for all this stuff is energy per bit transferred. Between chips on the same board, the channel is clean enough for many slow, simple wires: NVLink-C2C, the link between a Grace CPU and its Hopper GPU, has a forwarded clock, and moves 900 GB/s for 1.3 pJ/bit. NVLink between GPUs uses the SerDes treatment described above, at around 4 pJ/bit: at Blackwell’s 1.8 TB/s of NVLink bandwidth, roughly 58 W per GPU spent moving bits off the package. Past passive copper, a retimer at each cable end takes it to around ten, and optics to nearer twenty (the price of optics is the conversion from electrons to light, and is mostly indifferent to the distance travelled) 1.
Energy per bit against reach, one bar per link technology. These are quite fuzzy, but the direction is correct.
The binding constraint is where that energy cost is paid. Every joule the links expend is a joule inside the rack’s power and cooling envelope, which competes with compute. For now that means passive copper, and therefore scale-up stays within a rack.
How we got here & where we’re going #
Reading another chip’s memory with an ordinary load is an old idea for CPUs. SGI’s NUMAlink stitched as many as 512 processors into a single memory space by 2001, and coherent HyperTransport did the same for the handful of sockets on an Opteron board. Commodity clusters killed the big machines, but the board-scale version is everywhere: every multi-socket server is still a small
NUMA machine. On the GPU side, NVIDIA added a single virtual address space across GPUs, with peer-to-peer access over PCIe, in CUDA 4.0 (2011)CUDA 4.0 introduced unified virtual addressing and GPUDirect peer-to-peer on Fermi. Whether two GPUs can map each other’s memory depends on the bus between them; over PCIe it also requires both devices to sit under the same root complex.. By 2016 a gen3 x16 slot moved 16 GB/s per direction against 720 GB/s of local HBM2: one forty-fifth the bandwidth. NVLink was introduced with the P100 that year to try to widen the path: four links per GPU, with 160 GB/s of aggregate bidirectional bandwidth. The first generation also joined CPUs to GPUs: IBM sold POWER8 servers with [the CPUs wired to P100s
over NVLink](https://developer.nvidia.com/blog/inside-pascal/). Four links can’t fully connect eight GPUs, so the DGX-1 wired its P100s into a hybrid cube mesh, each GPU directly linked to four of the others. Transfers took one hop or two depending on the pair, and NCCL threaded its rings around the physical wiring. In 2018 NVIDIA put a switch between the GPUs instead: NVSwitch, an 18-port chip that carries traffic between all of its ports at the full link rate simultaneously. The DGX-2 holds twelve of them, and any pair of its sixteen V100s can transfer at the full 300 GB/s, with all sixteen GPUs communicating at once.
The third-generation switch (with Hopper, in 2022) carried ALUs for in-network reductionNVIDIA’s name for this is SHARP, the Scalable Hierarchical Aggregation and Reduction Protocol. It came over with the 2020 Mellanox acquisition: Mellanox’s InfiniBand switches had been doing in-network reductions for MPI collectives since 2016.: in an allreduce, every GPU sends its data into the switch once and receives the sum once, instead of shuttling partial results around a ring. That halves the bytes each GPU moves, and roughly doubles the effective bandwidth of the collective.
The GH200, in production since mid-2023, brings the CPU into the domain. It’s a Grace CPU paired with a Hopper GPU over NVLink-C2C, with the CPU’s 120 GB of LPDDR5X and the GPU’s 96 GB of HBM in one address space.
The GB200 followed in 2024, with one Grace for every two Blackwell GPUs. The NVL72 is 36 of them wired into a rack, the 72 GPUs and 36 CPUs joined through eighteen switch chips and a copper spine, 130 TB/s of any-to-any bandwidth between them. NVIDIA calls the rack “a single, massive GPU”. A thread on any of the 72 can issue a load against any byte of its 13.4 TB of HBM.
The link generations so farPair counts are per direction; per-GPU figures are NVIDIA’s bidirectional aggregates.:
| gen | GPU | year | per pair | pairs per link | links | per GPU |
|---|---|---|---|---|---|---|
| 1 | P100 | 2016 | 20 Gb/s NRZ | 8 | 4 | 160 GB/s |
| 2 | V100 | 2017 | 25 Gb/s NRZ | 8 | 6 | 300 GB/s |
| 3 | A100 | 2020 | 50 Gb/s NRZ | 4 | 12 | 600 GB/s |
| 4 | H100 | 2022 | 100 Gb/s PAM4 | 2 | 18 | 900 GB/s |
| 5 | B200 | 2024 | 200 Gb/s PAM4 | 2 | 18 | 1,800 GB/s |
NVIDIA has also named the next three stepsFrom NVIDIA’s Vera Rubin material and its GTC 2026 roadmap. Blank cells are numbers NVIDIA hasn’t yet put against the names.:
| NVLink | system | promised | per pair | per GPU | domain |
|---|---|---|---|---|---|
| 6 | Vera Rubin NVL72 | 2026 | 200 Gb/s PAM4, both directions | 3.6 TB/s | 72 GPUs, one rack |
| 7 | Rubin Ultra NVL576 | 2027 | — | — | 576 GPUs, eight racks |
| 8 | Feynman | 2028 | — | — | 1,152 GPUs, eight Kyber racks, optical between racks |
Vera Rubin keeps the rack as it is — 72 GPUs, a spine of five thousand copper cables — and doubles the bandwidth through it, to 260 TB/s in aggregate. The tick rate and the modulation are unchanged from NVLink 5, and the spine has no room for more cables. The doubling comes from each pair carrying both directions at once. Both ends transmit onto the same two wires at the same time, so the voltage at each receiver is the sum of two signals, one of which, its own, the receiver already knows. It synthesizes a copy of what its transmitter is sending and subtracts it; what remains is the far end’s signal. The subtraction is a timing problem: a tick lasts under ten picoseconds, and the copy has to line up with the actual echo to within a fraction of that, or the residue swamps the far end’s signalFrom SemiAnalysis’s Vera Rubin analysis; NVIDIA’s own material stops at the aggregate numbers. Gigabit Ethernet has driven its four twisted pairs this way since 1999, at a tick rate nearly a thousand times slower..
The NVL576 is the first NVLink domain bigger than a single rack: eight of them, with copper and optical links between the racksNVIDIA’s words: eight MGX NVL racks combined “with copper and direct optical connections”.. Kyber is the rack design that succeeds the 72-GPU one, the one the Feynman-era NVL1152 is built from. It fits 144 GPUs in one cabinet, and swaps the cable spine for a midplane, one large circuit board that the GPU trays plug into from both sides, the links running as traces inside it. In July 2026 SemiAnalysis reported that the board, at 78 layers, was proving hard to manufacture, and that the rack had slipped to 2028Reported via Tom’s Hardware, which also reports the NVL72x2 stopgap rack cancelled and NVL576 volumes in doubt.. NVIDIA’s response was that the roadmap is intact.
Other sides of the scale-up story #
Memory semantics are wired into NVIDIA’s answer to scale-up. But they’re not everyone’s. Google demonstrates that compiler-scheduled communication can create a tightly coupled scale-up system by another route.
Google connects its TPUs directly to one another, with no switch between them. Its network is a torus (most recently, a 64-chip rack arranged as a 4×4×4 cube), with 1.2 TB/s of ICI (inter-chip interconnect) per chipThe figures are Ironwood’s, the seventh generation, which scales to 9,216 chips. The torus dates to TPU v2’s 16×16 grid in 2017; the third dimension and the optical switching arrived with TPU v4.. Unlike NVIDIA’s model, a TPU doesn’t have an instruction to load from another chip’s memory. Communication is written explicitly into the program, and XLA schedules the collective operations alongside the computation at compile time. The participants and tensor shapes of a training step’s collectives are known before the program runs, and XLA uses that foreknowledge to overlap communication with compute and keep the direct links busy.
Between racks, the cubes are joined by optical circuit switches: mirrors that hold a light path open from one cube to another. Google repositions the mirrors between jobs, giving each job a torus of its own size and mapping out failed racksThe TPU v4 paper gives the reasons: cubes enter service before the whole pod exists, and a job can request variant topologies, including twisted tori for all-to-all traffic..
In an MoE model at inference time, the expert layers send each token to different chips chosen at runtime. On a torus and some reasonable MoE domain it can end up crossing some sixteen hops before finding its expert. The eighth TPU generation, announced in April 2026, forked the topology choices. TPU 8t, for pre-training, keeps the torus and grows it to 9,600 chips. TPU 8i, for inference, replaces the torus with “Boardfly”, 32-chip groups fully connected by copper, up to 36 groups joined through the same optical circuit switches, and any pair of the pod’s 1,024 active chips within seven hopsGoogle’s technical deep dive. The name is after Dragonfly, the 2008 high-radix topology it adapts..
On the open side of runtime dynamism: AMD, Broadcom, Intel, Google, Meta, Microsoft, and others formed the UALink consortium in 2024 to specify an open equivalent of NVLink. The 1.0 specification, published in April 2025, defines 200 Gb/s lanes, domains of up to 1,024 accelerators, and reads, writes, and atomics against remote memory in transactions of 64 to 256 bytes. But no UALink switch has shipped as of mid-2026. The first parts are due in labs late this year, and in products in 2027. In April 2026 the consortium published a 2.0 specification[The Register, April
2026](https://www.theregister.com/2026/04/07/ualink_2_specs/); the first
switch vendors, [Upscale AI and Astera
Labs](https://www.hpcwire.com/2025/12/02/upscale-ai-eyes-late-2026-for-scale-up-ualink-switch/),
both target late 2026.. AMD’s Helios racks (72 MI400-series GPUs, due in the second half of 2026) will carry UALink tunnelled through Broadcom Tomahawk 6 Ethernet switches in the meantimeReported by SemiAnalysis and The Next Platform. AMD quotes 260 TB/s of scale-up bandwidth for the rack, the same figure NVIDIA quotes for Vera Rubin..
There’s also an effort that parallels the rise of RoCE, an attempt to use Ethernet to dethrone a proprietary fabric. In July 2025 Broadcom shipped Tomahawk Ultra, a 51.2 Tb/s Ethernet switch chip aimed at NVSwitch. It is pin-compatible with the Tomahawk 5 that runs ordinary datacenter networks, with the Ethernet protocol
modified for scale-up: cut-down frame headers, link-layer retries, lossless
credit-based flow control (à la InfiniBand), and collectives executed inside the switch[ServeTheHome’s coverage](https://www.servethehome.com/broadcom-tomahawk-ultra-launch-for-scale-up-ethernet/)
has the details. The in-switch collectives include AllReduce and Broadcast: the SHARP feature set, in Ethernet silicon.. Port to port, it takes 250 nanoseconds. Broadcom had been in the group that announced UALink in May 2024, and did not take a seat when the consortium’s board formed that OctoberThe May 2024 group was AMD, Broadcom, Cisco, Google, HPE, Intel, Meta, and Microsoft. The October 2024 incorporation lists a board of AMD, Astera Labs, AWS, Cisco, Google, HPE, Intel, Meta, and Microsoft. Broadcom remains an ordinary member; its stated position, per The Register, is that Ethernet can do the job now rather than waiting for the UALink protocol.. In October 2025 the Open Compute Project opened a workstream, ESUN, to standardize Ethernet framing and switching for scale-up, seeded with
Broadcom’s [Scale-Up
Ethernet](https://www.broadcom.com/blog/tomahawk-ultra-ethernet-to-the-rescue-for-hpc-and-ai-scale-up)
specification. Its twelve founding members include both Broadcom and NVIDIAThe full list: AMD, Arista, Arm, Broadcom, Cisco, HPE, Marvell, Meta, Microsoft, NVIDIA, OpenAI, and Oracle..
With NVLink Fusion, announced in May 2025, NVIDIA is trying to license access to the NVLink fabric. Custom CPUs attach to NVIDIA GPUs over NVLink-C2C; or custom accelerators integrate an NVLink chiplet and join the switched domain. Fujitsu, Qualcomm, and Intel have agreed to attach CPUs; AWS’s Trainium4 will take the accelerator path, joining the NVLink 6 domain through the same switch trays as Vera RubinThe Fusion announcement, the Intel agreement, and [the Trainium4
integration](https://developer.nvidia.com/blog/aws-integrates-ai-infrastructure-with-nvidia-nvlink-fusion-for-trainium4-deployment/)..
Who wins? #
The previous post was about the scale-out version of this contest: InfiniBand against RoCE, a proprietary integrated fabric against commodity Ethernet adapted to the job, and the commodity side continues to win ground on cost and ubiquity. The scale-up battles have now started.
Unlike scale-out, there’s no clear commodity candidate to sweep the hyperscaler market and dethrone NVIDIA. NVLink is one of NVIDIA’s strongest assets: it’s taken them to rack-scale in a tightly integrated, all-to-all, ultra fast fabric years before anyone else. They got there by betting early and betting hard on integration. This makes it much more difficult for anyone else to push in and rebuild scale-up from the ground up & sell it to customers. Over the next few years, we’ll start to find out if scale-up is really so different.
Footnotes #
The only first-party number is C2C’s: NVIDIA quotes 1.3 pJ/bit in its
Grace Hopper deep dive. The single-ended, forwarded-clock design is described in NVIDIA’sISSCC 2023 paper. Nobody publishes one for NVLink itself; the ~4 pJ/bit isMarvell’s figurefor their long-reach 224G SerDes, the class of part these links are built from. The last two rungs are worked back from the power draw of shipping 800G parts: Credo’sproduct brieffor its 800G AEC specifies a typical 10 W dissipation, and lengths up to 7 m — over 800 Gb/s, about 12 pJ/bit; a pluggable optical module burns 13—18 W, 16—22 pJ/bit.↩
Suggest an edit Last modified: 21 Jul 2026