GPU FINGERPRINTING
Two A40s off the same line share a design, not silicon, and the order their atomic adds land in gives them away.
Your GPU has a tell #
Rent two A40s from the same datacenter and, on paper, you've got the same chip twice. Same model, same driver, same rack. Turns out they aren't the same at all: each one has its own small quirks, and those quirks leak into the numbers it computes. I built a way to tell physically identical GPUs apart using nothing but those numbers, and it works almost every time: 99.1% across 19 identical A40s, and 100% across seven identical RTX 3090s. It works on the 4090 and L40S too.
The fun part is that you don't even need a special kernel for it. Feed a plain torch.index_add_ call the right input, and it hands you the chip's identity right in its output tensor. People have fingerprinted GPUs before with custom-built race kernels,1<sup>1</sup> Hohentanner et al., "Unveiling Privacy Risks in WebGPU through Hardware-based Device Fingerprinting" (WiSec 2025), and Tee and Happel, "GPU Fingerprinting for Location Verification" (2026). but to my knowledge, nobody has identified individual GPUs from a standard PyTorch op. The fingerprint also survives a reboot (100% re-identification), and when I tried to make one chip fake another chip's fingerprint, it couldn't get anywhere close.
All of this cost me a few dollars of rented GPU time.
This post goes through how it works, how well it works, where it stops working, and how to make your GPU forgettable if you'd rather it stayed anonymous. The code and every single measurement are on GitHub.
Floating-point maths has a seating order #
Floating-point addition isn't associative: (a + b) + c and a + (b + c) can give you different answers, because every addition rounds.2<sup>2</sup> A float32 only keeps about 7 significant digits, so every addition rounds the result to the nearest number it can represent. Add a huge number and a tiny one, and the tiny one can vanish completely. Add the tiny ones together first, and they might survive. Order changes the answer. Normally the difference is tiny and nobody cares.
GPUs make this a lot more interesting. When thousands of threads all add into the same memory location with an atomic add,3<sup>3</sup> An atomic add is a read-add-write that the hardware guarantees nobody else can interrupt halfway. Many threads can safely add into the same location at once, but the hardware decides the order it serves them in. the hardware decides who goes first. You don't control that order, CUDA makes no promises about it, and it changes from run to run. That's why PyTorch labels some of its ops as "nondeterministic" on GPU.
Here's the thing though, it isn't actually random. The physical chip shapes the order: which streaming multiprocessor (SM) a block lands on, how far it sits from memory, and how each bit of silicon happens to time. Two A40s off the same line share a design, but no two dies come out of the fab exactly alike, so each one has its own slight, stable preference for who gets to add first. Average over enough runs, and that preference is the fingerprint.
The probe #
The first version is an active probe: a small CUDA kernel (written with CuPy) where every block races to atomically add into shared counters, and we record the order they arrive in. Run it a few times and you get each block's average arrival rank.
Now, most of that pattern is the same on every A40, because it comes from the architecture rather than the chip. Subtracting the average across all the chips gets rid of it, and whatever's left over belongs to that one specific chip. That leftover is the fingerprint. To identify a chip, you compare its leftover against the chips you've already enrolled, and pick the closest one.
I read it out two ways:
- Per-SM : group the arrival ranks by the physical SM each block ran on (CUDA exposes this through a register called
%smid). - Numerical only : just the ranks, which the kernel writes out as plain numbers, without SM IDs or timers. This is the stricter one, since it only uses what the kernel computes.
Does it work? #
Short answer: yes, scarily well.
- 19 identical A40s : the numerical-only fingerprint picked the right chip 111 times out of 112 (99.1% ), where guessing gets you 5.3%.
- 8 A40s in one batch :100% , with a permutation p-value of 0.0005.
- Same software, same result : four of those A40s had the exact same driver, CUDA toolchain and datacenter, and the fingerprint still picked the right one 100% of the time.
- It survives a reboot : I enrolled four chips, restarted their containers (fresh CUDA context, fresh install), and re-identified every one of them,12 out of 12 , with both readouts.
- It survives load : after a 60-second stress run, the per-SM fingerprint still matched every chip, 18 out of 18.
- It isn't an A40 quirk : the fingerprint told seven identical RTX 3090s apart 100% of the time (p = 0.0005), across four different driver versions. Pairs of RTX 4090s and L40Ss separated too, with the gap between chips 12 to 18 times bigger than the run-to-run noise.
If you score it like a biometric, the numerical fingerprint gets an AUC of 0.974 from a single measurement. Enrolling each chip with a few runs is what takes it up to 99%.
Can you fake it? #
Alright, but a fingerprint is only interesting if a chip can't just pretend to be a different chip. To test that, I played the attacker: sitting on chip A, trying to produce chip B's per-SM fingerprint using every knob a tenant actually controls, like grid size, block size, extra compute and injected delays. I tried ten configurations and ran each one twice.
- A chip matched its own fingerprint at a correlation of 0.946 on average, and never below0.916 .
- A different chip, with no forging attempt, matched at 0.174.
- The best forgery across every configuration I tried reached0.616 .
The best forgery stayed well below the lowest genuine score. Set the bar at 0.8, and every real chip gets in while the check turns away every forgery. Launch settings can nudge the arrival order a little, but nowhere near enough to copy another chip's.
Now do it with PyTorch #
This is the big one. The custom kernel works, but I wanted to know whether code people already run leaks the same fingerprint.
torch.index_add_ sums values into bins using atomics, which makes it one of PyTorch's nondeterministic ops, and it shows up all over the place: embedding gradients, graph neural net aggregation and MoE routing. My first attempt fed it ordinary random data and found absolutely nothing. With lots of values per bin, the order effects average out and the sum is just noise. I was about to write the PyTorch version off right there, but on more digging it turned out the input was the problem, not the op.
The trick is to make the order matter more than anything else. Give each bin only a few values, with magnitudes anywhere from 2^-30 to 2^30, so whichever value the GPU adds first decides how the rounding falls. Now the output itself carries the fingerprint:
import torch
K, PER = 65536, 4 # bins, values per bin
g = torch.Generator(device="cuda").manual_seed(20260831) # same input on every chip
idx = torch.arange(K, device="cuda").repeat_interleave(PER)
idx = idx[torch.randperm(K * PER, generator=g, device="cuda")]
e = torch.rand(K * PER, generator=g, device="cuda") * 60 - 30
s = torch.where(torch.rand(K * PER, generator=g, device="cuda") > 0.5, 1.0, -1.0)
src = (s * torch.exp2(e)).float() # magnitudes 2^-30 .. 2^30
ref = torch.zeros(K, dtype=torch.float64, device="cuda").index_add_(0, idx, src.double())
resid = torch.zeros(K, dtype=torch.float64, device="cuda")
for _ in range(400):
out = torch.zeros(K, device="cuda").index_add_(0, idx, src) # the real op
resid += out.double() - ref
signature = torch.sign(resid) # one bit per bin
That's the whole thing: the output of an op that ships with PyTorch, and not a single custom kernel anywhere.
- Chips on the same driver :100% (9 out of 9 across three A40s).
- Six A40s on mixed drivers : 78%, against 17% for guessing. The raw output picks up a little of the driver version as well as the chip.
- Three RTX 4090s : 6 out of 6, with an equal error rate of 0%.
- Capacity : about217 independent bits per A40 signature and 239 on the 4090. That's enough to tell apart a very, very large fleet.
Where it stops #
So can you work out which GPU served your chatbot reply from its logits? No.
I ran a 6-layer transformer forward and backward 200 times each on three A40s, in both normal and deterministic mode. The logits and gradients were bit-identical from run to run and across all three chips. Dense matmuls and TF32 matmuls behaved the same way, and with no variation, there's nothing to fingerprint. Clock speeds and power draw under sustained load didn't separate the chips either.
The leak needs atomics adding into the same place, in an order that changes the answer. Standard transformer inference avoids that, while scatter-heavy ops like index_add_ rely on it.
Witness protection for your GPU #
If you'd rather your GPU stayed anonymous, here's what works:
- Turn on deterministic mode.
torch.use_deterministic_algorithms(True)(plusCUBLAS_WORKSPACE_CONFIG=:4096:8) made theindex_add_output bit-identical across chips, and the PyTorch fingerprint vanished completely. It isn't free though, backward passes ran 2.5 to 3.6 times slower, and forward passes about 1.2 times. - The custom probe doesn't care. It's the tenant's own CUDA kernel, so PyTorch's flags never touch it. Stopping it means not letting people run custom kernels, hiding the
%smidregister, or virtualising how the GPU schedules work onto SMs, all of which are a lot heavier.
The PyTorch leak is cheap to close, then, while the custom-kernel probe is very hard to close.
What it's good for #
Cloud providers sell GPUs as interchangeable, but a tenant can tell exactly which physical chip they got. That's handy for checking you're actually on the hardware you paid for, keeping an inventory of a fleet without trusting what the host reports, or tracing which chip produced a result. The flip side is privacy: a tenant can also tell when they've landed on the same chip again.
Next up, compute permitting, I want to run this on dozens of chips per model to put a proper number on collisions, try it on H100s and B200s, and go looking for more everyday ops that leak the same way, like MoE dispatch and embedding bags.
Try it #
The repo includes every measurement from this post, so you can reproduce the numbers on a CPU without touching a GPU:
git clone https://github.com/divagr18/gpu-instance-fingerprint && cd gpu-instance-fingerprint
pip install -r requirements.txt
python analysis/pooled_analyze.py # 19 A40s: 111/112 = 99.1%
python analysis/analyze_puf.py "a40_8chip/*.json" # 8 A40s: 32/32 = 100%
To fingerprint your own GPUs, you'll need CuPy for the probe, or just PyTorch for the index_add_ version. Collect a few runs from two or more GPUs of the same model, then point the analysis at them:
pip install numpy cupy-cuda12x
python probes/probe_puf.py 3 0 > gpu_a_r0.json # 3 epochs, run 0
python probes/probe_puf.py 3 1 > gpu_a_r1.json
python analysis/analyze_puf.py "path/to/*.json"
python probes/passive_index_add.py 0 65536 > passive_r0.json # the PyTorch version
If you use it in your work, you can cite it as:
@software{agrawal2026gpufingerprint,
author = {Agrawal, Divyansh},
title = {Telling identical {GPUs} apart from their numerical output},
year = {2026},
url = {https://github.com/divagr18/gpu-instance-fingerprint}
}
Notes #
- Hohentanner et al., "Unveiling Privacy Risks in WebGPU through Hardware-based Device Fingerprinting " (WiSec 2025), and Tee and Happel, "GPU Fingerprinting for Location Verification " (2026).↩
- A float32 only keeps about 7 significant digits, so every addition rounds the result to the nearest number it can represent. Add a huge number and a tiny one, and the tiny one can vanish completely. Add the tiny ones together first, and they might survive. Order changes the answer. ↩
- An atomic add is a read-add-write that the hardware guarantees nobody else can interrupt halfway. Many threads can safely add into the same location at once, but the hardware decides the order it serves them in. ↩