# NVLink, NVSwitch, and all that

> Source: <https://fergusfinn.com/blog/nvlink-scale-up/>
> Published: 2026-07-21 00:00:00+00:00

# NVLink, NVSwitch, and all that

I wrote [before](https://fergusfinn.com/blog/infiniband-roce-rdma/) about
‘scale-out’ fabrics — [InfiniBand](https://en.wikipedia.org/wiki/InfiniBand),
[RoCE](https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet), and more
generally
[RDMA](https://en.wikipedia.org/wiki/Remote_direct_memory_access) — the networks and paradigms that stitch thousands of machines into a datacenter
island.

This post is about the other kind. The names come from an old argument about
servers: when you need more computer, you can either buy more machines and
*scale out*, or buy a bigger machine and *scale up*. A scale-up fabric is an
attempt to connect processors so tightly that they can behave as one. Inside a
modern (NVIDIA) GPU machine the fabric is
[ NVLink](https://en.wikipedia.org/wiki/NVLink)The terms are a bit frustrating really. Because you can scale up your LLM
inference domain over your scale-out network. And you can certainly scale out inside a
scale-up domain..

## Fast, or far

Let’s follow the physics of sending bytes over a link. At the lowest level: an
NVLink is a bundle of wires, set up as [differential
pairs](https://en.wikipedia.org/wiki/Differential_signalling). Each pair is
driven by a [SerDes](https://en.wikipedia.org/wiki/SerDes): a circuit that
turns wide, slow on-chip data into one very fast signal on the wire.

The signal on the wire is just a voltage changing over time. To turn it back into zeros and ones the receiver has to sample it at the right instants. Before deciding what to put on the wire, we have to decide where the receiver’s sense of time is going to come from.

Length-matching meanders next to a BGA package
([source](https://i.redd.it/ys3gywkwkr7g1.jpeg)).

One answer is to send a clock pulse, separate to the data, on its own wire, and
sample as the clock ticks (a *forwarded clock*). This is how [DDR
memory](https://en.wikipedia.org/wiki/DDR_SDRAM) and
[HBM](https://en.wikipedia.org/wiki/High_Bandwidth_Memory) work. But it only
works if the clock’s ticks arrive close enough to all the signal bits for the
receiver to sample them cleanly. The receiver can deskew small differences, but
every data wire still needs about the same delay and, in practice, about the
same length. This is why PCBs are covered in
squiggles. At a hundred gigabits per second, one bit occupies about ten
picoseconds: a millimetre or two of trace. That kind of matching is possible
across centimetres, and gets much harder across metres.

We can instead synchronize each wire individually, sending its clock over each
signal wire itself. We know that every transition in the signal (each 0-to-1 or
1-to-0) marks a bit boundary. So we can use a circuit at the receiver ([clock
and data recovery](https://en.wikipedia.org/wiki/Clock_recovery)) that keeps a
local oscillator locked to those transitions. The problem is that it works only
if the transitions keep coming. If you send a million zeros in a row, the
oscillator can start to drift. To keep the data transitioningStatistically, not provably. In principle you can craft data that XORs to
a long run of zeros (“killer packets”, seen in
[SONET](https://en.wikipedia.org/wiki/Synchronous_Optical_Networking)). Modern
links use long scrambler polynomials to make this impractical., we XOR it
with a pseudo-random sequence that both ends can generate
([scrambling](https://en.wikipedia.org/wiki/Scrambler)).

Top: each edge in the data re-centres the receiver’s clock, but across the run of zeros it free-runs (drawn a few percent fast) and the samples slide off-centre until one lands in the wrong bit. Bottom: the same payload scrambled, the edges never stop.

Once we can carry a timing signal, we need to put our 1s and 0s on the wire.
One of the simplest ways is: high
voltage for a 1, low for a 0. That scheme is called
[NRZ](https://en.wikipedia.org/wiki/Non-return-to-zero)Non-return-to-zero: return-to-zero being a version where you return to
zero between every signalled bit.

To signal faster on a single wire under NRZ you have to tick faster. A trace on
a board works like a [low-pass
filter](https://en.wikipedia.org/wiki/Low-pass_filter): the copper, and the
insulation around it, both absorb more at higher frequenciesThe copper loses because high-frequency current crowds into a thin shell
at the conductor’s surface (the [skin
effect](https://en.wikipedia.org/wiki/Skin_effect)). The insulation loses
because the signal’s energy travels through it via the electromagnetic field,
and the material’s molecules absorb some of it as the field alternates
([dielectric
loss](https://en.wikipedia.org/wiki/Dielectric_loss))., adding
attenuation that the receiver has to make back. It does this with filters that
re-boost the high
end, but this boosting amplifies the high-frequency noise along with the
signal.

Rather than keep paying that rising attenuation, at some point it’s better to
try to send more bits per tick.
[PAM4](https://en.wikipedia.org/wiki/Pulse-amplitude_modulation) signals four
voltage levels instead of two, so each tick carries 2 bits of information. The
cost is that the levels sit a third as far apart. As the level spacing gets
closer, errors in decoding start to become more likely.

PAM4 links aim for roughly one bit in ten thousand wrong before correction,
where NRZ links without correction can aim for better than one in a trillion.
At these speeds an error rate of one in
a trillion means an error every ten seconds or so, which is rare enough you can
manage it at the link layer with checksums and resends. One in ten thousand
means millions of errors a second, which is far too many to handle by resending. So
PAM4 is usually paired with [forward error
correction](https://en.wikipedia.org/wiki/Error_correction_code). This costs
latency, since the decoder has to collect a whole block before it can correct
anything.

Instead of signalling furiously on a single wire, can’t you just add more
wires? Yes, up to a point. Fast signals have to leave from the die’s edge: a
SerDes buried mid-die would have to thread its traces out past all the stuff
around it. Off-chip bandwidth therefore scales with the die’s perimeter (its
**shoreline**) while compute scales with its area. The [reticle
limit](https://en.wikipedia.org/wiki/Photolithography) caps the die at
roughly 26 by 33 mm: about 120 mm of shoreline. The [HBM](https://en.wikipedia.org/wiki/High_Bandwidth_Memory)
interfaces, NVLink, PCIe, and C2C all compete for that finite edge.

The other variable is distance. Higher frequency signals travel less far:
passive copper carries 100 Gb/s per pair about two metres, and 200 Gb/s about
oneThese are IEEE spec objectives: IEEE 802.3ck specifies
100G-per-lane passive copper to at least 2 m; the follow-on
[P802.3dj](https://www.ieee802.org/3/dj/index.html) carries a 1 m objective at
200G per lane, and TE and Intel demoed 224G PAM4 over a 1 m passive cable at
[OFC 2023](https://opticalconnectionsnews.com/2023/03/ofc-2023-te-intel-demo-224g-pam4-over-passive-copper/).
Longer is possible with effort: [Marvell](https://www.marvell.com/blogs/224g-long-range-serdes-scale-up-scale-inside.html)
has driven 2.5 m of cumulative copper with a long-reach 224G SerDes.. To go further requires extra hardware steps. A *redriver* is an analog
amplifier, and a *retimer* a kind of digital one: a full receiver and
transmitter back-to-back decodes the bits and relaunches the signal anew.

The ledger for all this stuff is energy per bit transferred. Between
chips on the same board, the channel is clean enough for many slow, simple
wires: NVLink-C2C, the link between a Grace CPU and its Hopper GPU, has a
forwarded clock, and moves 900 GB/s for 1.3 pJ/bit. NVLink between GPUs
uses the SerDes treatment described above, at around 4 pJ/bit: at Blackwell’s
1.8 TB/s of NVLink bandwidth, roughly 58 W per GPU spent moving bits off the
package. Past passive copper, a retimer at each cable end takes it to around
ten, and optics to nearer twenty (the price of optics is the conversion from
electrons to light, and is mostly indifferent to the distance
travelled) 1.

Energy per bit against reach, one bar per link technology. These are quite fuzzy, but the direction is correct.

The binding constraint is where that energy cost is paid. Every joule the links expend is a joule inside the rack’s power and cooling envelope, which competes with compute. For now that means passive copper, and therefore scale-up stays within a rack.

## How we got here & where we’re going

Reading another chip’s memory with an ordinary load is an old idea for CPUs.
SGI’s [NUMAlink](https://en.wikipedia.org/wiki/NUMAlink) stitched as many as
512 processors into a single memory space by 2001, and coherent
[HyperTransport](https://en.wikipedia.org/wiki/HyperTransport) did the same for
the handful of sockets on an Opteron board. Commodity clusters killed the big
machines, but the board-scale version is everywhere: every multi-socket
server is still a small
[NUMA](https://en.wikipedia.org/wiki/Non-uniform_memory_access) machine.

On the GPU side, NVIDIA added a single virtual address space across GPUs, with
peer-to-peer access over PCIe, in CUDA 4.0 (2011)[CUDA 4.0](https://developer.nvidia.com/blog/unified-memory-in-cuda-6/)
introduced unified virtual addressing and GPUDirect peer-to-peer on Fermi.
Whether two GPUs can map each other’s memory depends on the bus between them;
over PCIe it also requires both devices to sit under the same root complex.. By 2016 a gen3 x16 slot
moved 16 GB/s per direction against 720 GB/s of local HBM2: one forty-fifth the
bandwidth. NVLink was
introduced with the P100 that year to try to widen the path: four links per
GPU, with 160 GB/s of aggregate bidirectional bandwidth. The first generation
also joined CPUs to GPUs: IBM sold POWER8 servers with [the CPUs wired to P100s
over NVLink](https://developer.nvidia.com/blog/inside-pascal/).

Four links can’t fully connect eight GPUs, so the DGX-1 wired its P100s into a
[hybrid cube mesh](https://developer.nvidia.com/blog/inside-pascal/), each GPU
directly linked to four of the others. Transfers took one hop or two depending
on the pair, and NCCL threaded its rings around the physical wiring. In 2018
NVIDIA put a switch between the GPUs instead:
[NVSwitch](https://developer.nvidia.com/blog/nvswitch-accelerates-nvidia-dgx2/),
an 18-port chip that carries traffic between all of its ports at the full
link rate simultaneously. The DGX-2 holds twelve of them, and any pair of its
sixteen V100s can transfer at the full 300 GB/s, with all sixteen GPUs
communicating at once.

The third-generation switch (with Hopper, in 2022) carried ALUs for in-network
reductionNVIDIA’s name for this is
[SHARP](https://docs.nvidia.com/networking/display/sharpv300), the Scalable
Hierarchical Aggregation and Reduction Protocol. It came over with the 2020
Mellanox acquisition: Mellanox’s InfiniBand switches had been doing in-network
reductions for MPI collectives since 2016.:
in an allreduce, every GPU sends its data into the switch once and receives
the sum once, instead of shuttling partial results around a ring. That halves
the bytes each GPU moves, and roughly doubles the effective bandwidth of the
collective.

The
[GH200](https://nvidianews.nvidia.com/news/nvidia-grace-hopper-superchips-designed-for-accelerated-generative-ai-enter-full-production),
in production since mid-2023, brings the CPU into the domain. It’s a Grace CPU paired with
a Hopper GPU over NVLink-C2C, with the CPU’s 120 GB of LPDDR5X and the GPU’s 96 GB of
HBM in one address space.

The GB200 followed in 2024, with one Grace for every two Blackwell GPUs. The
[NVL72](https://www.nvidia.com/en-us/data-center/gb200-nvl72/) is 36 of them
wired into a rack, the 72 GPUs and 36 CPUs joined through eighteen switch
chips and a copper spine, 130 TB/s of any-to-any bandwidth between them.
NVIDIA calls the rack “a single, massive GPU”. A thread on any of the 72 can
issue a load against any byte of its 13.4 TB of HBM.

The link generations so farPair counts are per direction; per-GPU figures are NVIDIA’s bidirectional aggregates.:

| gen | GPU | year | per pair | pairs per link | links | per GPU |
|---|---|---|---|---|---|---|
| 1 | P100 | 2016 | 20 Gb/s NRZ | 8 | 4 | 160 GB/s |
| 2 | V100 | 2017 | 25 Gb/s NRZ | 8 | 6 | 300 GB/s |
| 3 | A100 | 2020 | 50 Gb/s NRZ | 4 | 12 | 600 GB/s |
| 4 | H100 | 2022 | 100 Gb/s PAM4 | 2 | 18 | 900 GB/s |
| 5 | B200 | 2024 | 200 Gb/s PAM4 | 2 | 18 | 1,800 GB/s |

NVIDIA has also named the next three stepsFrom [NVIDIA’s Vera Rubin
material](https://developer.nvidia.com/blog/nvidia-vera-rubin-pod-seven-chips-five-rack-scale-systems-one-ai-supercomputer/)
and its GTC 2026 roadmap. Blank cells are numbers NVIDIA hasn’t yet put
against the names.:

| NVLink | system | promised | per pair | per GPU | domain |
|---|---|---|---|---|---|
| 6 | Vera Rubin NVL72 | 2026 | 200 Gb/s PAM4, both directions | 3.6 TB/s | 72 GPUs, one rack |
| 7 | Rubin Ultra NVL576 | 2027 | — | — | 576 GPUs, eight racks |
| 8 | Feynman | 2028 | — | — | 1,152 GPUs, eight Kyber racks, optical between racks |

Vera Rubin keeps the rack as it is — 72 GPUs, a spine of five thousand copper
cables — and doubles the bandwidth through it, to 260 TB/s in aggregate. The
tick rate and the modulation are unchanged from NVLink 5, and the spine has no
room for more cables. The doubling comes from each pair carrying both
directions at once.
Both ends transmit onto the same two wires at the same time, so the voltage at
each receiver is the sum of two signals, one of which, its own, the receiver
already knows. It synthesizes a copy of what its transmitter is sending and
subtracts it; what remains is the far end’s signal. The subtraction is a timing problem: a tick
lasts under ten picoseconds, and the copy has to line up with the actual echo
to within a fraction of that, or the residue swamps the far end’s
signalFrom [SemiAnalysis’s Vera Rubin
analysis](https://newsletter.semianalysis.com/p/vera-rubin-extreme-co-design-an-evolution);
NVIDIA’s own material stops at the aggregate numbers. Gigabit Ethernet has
driven its four twisted pairs this way since 1999, at a tick rate nearly a
thousand times slower..

The NVL576 is the first NVLink domain bigger than a single rack: eight of
them, with copper and optical links between the racksNVIDIA’s words: eight MGX NVL racks combined “with copper and direct
optical connections”.. Kyber is the rack
design that succeeds the 72-GPU one, the one the Feynman-era NVL1152 is built
from. It fits 144 GPUs in one cabinet, and
swaps the cable spine for a midplane, one large circuit board that the GPU
trays plug into from both sides, the links running as traces inside it. In
July 2026 SemiAnalysis reported that the board, at 78 layers, was proving
hard to manufacture, and that the rack had slipped to 2028Reported via [Tom’s
Hardware](https://www.tomshardware.com/pc-components/gpus/nvidias-kyber-rack-for-rubin-ultra-slips-to-2028),
which also reports the NVL72x2 stopgap rack cancelled and NVL576 volumes in
doubt.. NVIDIA’s
response was that the roadmap is intact.

## Other sides of the scale-up story

Memory semantics are wired into NVIDIA’s answer to scale-up. But they’re not everyone’s. Google demonstrates that compiler-scheduled communication can create a tightly coupled scale-up system by another route.

Google connects its TPUs directly to one another, with no switch between them.
Its network is a torus (most recently, a 64-chip rack arranged as a 4×4×4 cube),
with 1.2 TB/s of ICI (inter-chip interconnect) per chipThe figures are
[Ironwood](https://docs.cloud.google.com/tpu/docs/tpu7x)’s, the seventh
generation, which scales to 9,216 chips. The torus dates to TPU v2’s 16×16
grid in 2017; the third dimension and the optical switching arrived with [TPU
v4](https://arxiv.org/abs/2304.01433).. Unlike NVIDIA’s
model, a TPU doesn’t have an instruction to load from another chip’s memory.
Communication is written explicitly into the program, and
[XLA](https://openxla.org/xla) schedules the collective operations alongside
the computation at compile time. The participants and tensor shapes of a
training step’s collectives are known before the program runs, and XLA uses
that foreknowledge to overlap communication with compute and keep the direct
links busy.

Between racks, the cubes are joined by optical circuit switches: mirrors that
hold a light path open from one cube to another. Google repositions the mirrors
between jobs, giving each job a torus of its own size and mapping out failed
racksThe [TPU v4 paper](https://arxiv.org/abs/2304.01433) gives the
reasons: cubes enter service before the whole pod exists, and a job can
request variant topologies, including twisted tori for all-to-all traffic..

In an MoE model at inference time, the expert layers send each token to
different chips chosen at runtime. On a torus and some reasonable MoE domain it
can end up crossing some sixteen hops before finding its expert. The eighth TPU
generation, announced in April 2026, forked the topology choices. TPU 8t, for
pre-training, keeps the torus and grows it to 9,600 chips. TPU 8i, for
inference, replaces the torus with “Boardfly”, 32-chip groups fully connected
by copper, up to 36 groups joined through the same optical circuit switches,
and any pair of the pod’s 1,024 active chips within seven hops[Google’s technical deep
dive](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive).
The name is after
[Dragonfly](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/34926.pdf),
the 2008 high-radix topology it adapts..

On the open side of runtime dynamism: AMD, Broadcom, Intel, Google, Meta,
Microsoft, and others formed the [UALink consortium](https://ualinkconsortium.org/) in
2024 to specify an open equivalent of NVLink. The [1.0
specification](https://ualinkconsortium.org/blog/ualink-200g-1-0-specification-overview-802/),
published in April 2025, defines 200 Gb/s lanes, domains of up to 1,024
accelerators, and reads, writes, and atomics against remote memory in
transactions of 64 to 256 bytes. But no UALink switch has shipped as of
mid-2026. The first parts are due in labs late this year, and in products in
2027. In April 2026 the consortium published a 2.0 specification[The Register, April
2026](https://www.theregister.com/2026/04/07/ualink_2_specs/); the first
switch vendors, [Upscale AI and Astera
Labs](https://www.hpcwire.com/2025/12/02/upscale-ai-eyes-late-2026-for-scale-up-ualink-switch/),
both target late 2026.. AMD’s
Helios racks (72 MI400-series GPUs, due in the second half of 2026) will
carry UALink tunnelled through Broadcom Tomahawk 6 Ethernet switches in
the meantimeReported by [SemiAnalysis](https://newsletter.semianalysis.com/p/amd-advancing-ai-mi350x-and-mi400-ualoe72-mi500-ual256) and [The Next Platform](https://www.nextplatform.com/compute/2026/02/23/amd-says-helios-racks-and-mi400-series-gpus-on-track-for-2h-2026/4092199).
AMD quotes 260 TB/s of scale-up bandwidth for the rack, the same figure
NVIDIA quotes for Vera Rubin..

There’s also an effort that parallels the rise of RoCE, an attempt to use
Ethernet to dethrone a proprietary fabric. In
July 2025 Broadcom shipped [Tomahawk
Ultra](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-ultra-reimagining-ethernet-switch-hpc),
a 51.2 Tb/s Ethernet switch chip aimed at NVSwitch. It is pin-compatible with
the Tomahawk 5 that runs ordinary datacenter networks, with the Ethernet protocol
modified for scale-up: cut-down frame headers, link-layer retries, lossless
credit-based flow control (à la InfiniBand), and collectives executed inside the switch[ServeTheHome’s coverage](https://www.servethehome.com/broadcom-tomahawk-ultra-launch-for-scale-up-ethernet/)
has the details. The in-switch collectives include AllReduce and
Broadcast: the SHARP feature set, in Ethernet silicon..
Port to port, it takes 250 nanoseconds. Broadcom had been in the group that
announced UALink in May 2024, and did not take a seat when the consortium’s
board formed that OctoberThe [May 2024 group](https://www.businesswire.com/news/home/20240530653602/en/AMD-Broadcom-Cisco-Google-Hewlett-Packard-Enterprise-Intel-Meta-and-Microsoft-Form-Ultra-Accelerator-Link-UALink-Promoter-Group-to-Drive-Data-Center-AI-Connectivity)
was AMD, Broadcom, Cisco, Google, HPE, Intel, Meta, and Microsoft. The
[October 2024 incorporation](https://www.businesswire.com/news/home/20241029998800/en/Ultra-Accelerator-Link-Consortium-Incorporates-Announces-Membership-Opportunity)
lists a board of AMD, Astera Labs, AWS, Cisco, Google, HPE, Intel, Meta, and
Microsoft. Broadcom remains an ordinary member; its stated position, [per The
Register](https://www.theregister.com/2025/07/15/broadcom_ethernet_scale_up/),
is that Ethernet can do the job now rather than waiting for the UALink
protocol.. In October 2025 the Open Compute Project opened
a workstream,
[ESUN](https://www.opencompute.org/blog/introducing-esun-advancing-ethernet-for-scale-up-ai-infrastructure-at-ocp),
to standardize Ethernet framing and switching for scale-up, seeded with
Broadcom’s [Scale-Up
Ethernet](https://www.broadcom.com/blog/tomahawk-ultra-ethernet-to-the-rescue-for-hpc-and-ai-scale-up)
specification. Its twelve founding members include both Broadcom and
NVIDIAThe full list: AMD, Arista, Arm, Broadcom, Cisco, HPE, Marvell, Meta,
Microsoft, NVIDIA, OpenAI, and Oracle..

With [NVLink
Fusion](https://nvidianews.nvidia.com/news/nvidia-nvlink-fusion-semi-custom-ai-infrastructure-partner-ecosystem),
announced in May 2025, NVIDIA is trying to license access to the NVLink fabric.
Custom CPUs attach to NVIDIA GPUs over NVLink-C2C; or custom accelerators
integrate an NVLink chiplet and join the switched domain. Fujitsu, Qualcomm, and Intel
have agreed to attach CPUs; AWS’s Trainium4 will take the accelerator path,
joining the NVLink 6 domain through the same switch trays as Vera Rubin[The Fusion
announcement](https://www.tomshardware.com/pc-components/cpus/nvidia-announces-nvlink-fusion-to-allow-custom-cpus-and-ai-accelerators-to-work-with-its-products),
[the Intel
agreement](https://nvidianews.nvidia.com/news/nvidia-and-intel-to-develop-ai-infrastructure-and-personal-computing-products),
and [the Trainium4
integration](https://developer.nvidia.com/blog/aws-integrates-ai-infrastructure-with-nvidia-nvlink-fusion-for-trainium4-deployment/)..

## Who wins?

The previous post was about the scale-out version of this contest:
[InfiniBand](https://fergusfinn.com/blog/infiniband-roce-rdma/) against RoCE,
a proprietary integrated fabric against commodity Ethernet adapted to the
job, and the commodity side continues to win ground on cost and ubiquity. The
scale-up battles have now started.

Unlike scale-out, there’s no clear commodity candidate to sweep the hyperscaler market and dethrone NVIDIA. NVLink is one of NVIDIA’s strongest assets: it’s taken them to rack-scale in a tightly integrated, all-to-all, ultra fast fabric years before anyone else. They got there by betting early and betting hard on integration. This makes it much more difficult for anyone else to push in and rebuild scale-up from the ground up & sell it to customers. Over the next few years, we’ll start to find out if scale-up is really so different.

## Footnotes

-
The only first-party number is C2C’s: NVIDIA quotes 1.3 pJ/bit in its

[Grace Hopper deep dive](https://developer.nvidia.com/blog/nvidia-grace-hopper-superchip-architecture-in-depth/). The single-ended, forwarded-clock design is described in NVIDIA’s[ISSCC 2023 paper](https://ieeexplore.ieee.org/document/10067395/). Nobody publishes one for NVLink itself; the ~4 pJ/bit is[Marvell’s figure](https://www.marvell.com/blogs/224g-long-range-serdes-scale-up-scale-inside.html)for their long-reach 224G SerDes, the class of part these links are built from. The last two rungs are worked back from the power draw of shipping 800G parts: Credo’s[product brief](https://credosemi.com/wp-content/uploads/Credo_AEC_800G_SPAN_O_O_RHS_Product_Brief_2025Sept26.pdf)for its 800G AEC specifies a typical 10 W dissipation, and lengths up to 7 m — over 800 Gb/s, about 12 pJ/bit; a pluggable optical module burns 13—18 W, 16—22 pJ/bit.[↩](#user-content-fnref-1)

[Suggest an edit](https://github.com/fergusfinn/blog/edit/main/src/content/blog/nvlink-scale-up.mdx)

Last modified: 21 Jul 2026