Telling identical GPUs apart with one line of PyTorch

A GPU fingerprinting technique using a standard PyTorch `torch.index_add_` call identified individual physically identical GPUs with 99.1% accuracy across 19 identical Nvidia A40s and 100% accuracy across seven identical RTX 3090s, according to the author's published results and GitHub code. The method exploits the nondeterministic order in which atomic adds land in memory, which the author says is shaped by stable per-die silicon quirks; the fingerprint survived a reboot with 100% re-identification, and an attempt to make one chip imitate another's fingerprint failed. The author reports the work cost a few dollars of rented GPU time and says it is the first identification of individual GPUs from a standard PyTorch op, extending to the RTX 4090 and L40S.

← My writing https://divagr.com/blog GPU FINGERPRINTING Telling identical GPUs apart with one line of PyTorch Two A40s off the same line share a design, not silicon, and the order their atomic adds land in gives them away. Your GPU has a tell Rent two A40s from the same datacenter and, on paper, you've got the same chip twice. Same model, same driver, same rack. Turns out they aren't the same at all: each one has its own small quirks, and those quirks leak into the numbers it computes. I built a way to tell physically identical GPUs apart using nothing but those numbers, and it works almost every time: 99.1% across 19 identical A40s , and 100% across seven identical RTX 3090s . It works on the 4090 and L40S too. The fun part is that you don't even need a special kernel for it. Feed a plain torch.index add call the right input, and it hands you the chip's identity right in its output tensor. People have fingerprinted GPUs before with custom-built race kernels, 1 footnote-1 <sup 1</sup Hohentanner et al., " Unveiling Privacy Risks in WebGPU through Hardware-based Device Fingerprinting https://doi.org/10.1145/3734477.3734700 " WiSec 2025 , and Tee and Happel, " GPU Fingerprinting for Location Verification https://arxiv.org/abs/2605.01930 " 2026 . but to my knowledge, nobody has identified individual GPUs from a standard PyTorch op . The fingerprint also survives a reboot 100% re-identification , and when I tried to make one chip fake another chip's fingerprint, it couldn't get anywhere close. All of this cost me a few dollars of rented GPU time. This post goes through how it works, how well it works, where it stops working, and how to make your GPU forgettable if you'd rather it stayed anonymous. The code and every single measurement are on GitHub https://github.com/divagr18/gpu-instance-fingerprint . Floating-point maths has a seating order Floating-point addition isn't associative: a + b + c and a + b + c can give you different answers, because every addition rounds. 2 footnote-2 <sup 2</sup A float32 only keeps about 7 significant digits, so every addition rounds the result to the nearest number it can represent. Add a huge number and a tiny one, and the tiny one can vanish completely. Add the tiny ones together first, and they might survive. Order changes the answer. Normally the difference is tiny and nobody cares. GPUs make this a lot more interesting. When thousands of threads all add into the same memory location with an atomic add, 3 footnote-3 <sup 3</sup An atomic add is a read-add-write that the hardware guarantees nobody else can interrupt halfway. Many threads can safely add into the same location at once, but the hardware decides the order it serves them in. the hardware decides who goes first. You don't control that order, CUDA makes no promises about it, and it changes from run to run. That's why PyTorch labels some of its ops as "nondeterministic" on GPU. Here's the thing though, it isn't actually random. The physical chip shapes the order: which streaming multiprocessor SM a block lands on, how far it sits from memory, and how each bit of silicon happens to time. Two A40s off the same line share a design, but no two dies come out of the fab exactly alike, so each one has its own slight, stable preference for who gets to add first. Average over enough runs, and that preference is the fingerprint. The probe The first version is an active probe: a small CUDA kernel written with CuPy where every block races to atomically add into shared counters, and we record the order they arrive in. Run it a few times and you get each block's average arrival rank. Now, most of that pattern is the same on every A40, because it comes from the architecture rather than the chip. Subtracting the average across all the chips gets rid of it, and whatever's left over belongs to that one specific chip. That leftover is the fingerprint. To identify a chip, you compare its leftover against the chips you've already enrolled, and pick the closest one. I read it out two ways: - Per-SM : group the arrival ranks by the physical SM each block ran on CUDA exposes this through a register called %smid . - Numerical only : just the ranks, which the kernel writes out as plain numbers, without SM IDs or timers. This is the stricter one, since it only uses what the kernel computes. Does it work? Short answer: yes, scarily well. - 19 identical A40s : the numerical-only fingerprint picked the right chip 111 times out of 112 99.1% , where guessing gets you 5.3%. - 8 A40s in one batch : 100% , with a permutation p-value of 0.0005. - Same software, same result : four of those A40s had the exact same driver, CUDA toolchain and datacenter, and the fingerprint still picked the right one 100% of the time. - It survives a reboot : I enrolled four chips, restarted their containers fresh CUDA context, fresh install , and re-identified every one of them, 12 out of 12 , with both readouts. - It survives load : after a 60-second stress run, the per-SM fingerprint still matched every chip, 18 out of 18. - It isn't an A40 quirk : the fingerprint told seven identical RTX 3090s apart 100% of the time p = 0.0005 , across four different driver versions. Pairs of RTX 4090s and L40Ss separated too, with the gap between chips 12 to 18 times bigger than the run-to-run noise. If you score it like a biometric, the numerical fingerprint gets an AUC of 0.974 from a single measurement. Enrolling each chip with a few runs is what takes it up to 99%. Can you fake it? Alright, but a fingerprint is only interesting if a chip can't just pretend to be a different chip. To test that, I played the attacker: sitting on chip A, trying to produce chip B's per-SM fingerprint using every knob a tenant actually controls, like grid size, block size, extra compute and injected delays. I tried ten configurations and ran each one twice. - A chip matched its own fingerprint at a correlation of 0.946 on average, and never below 0.916 . - A different chip, with no forging attempt, matched at 0.174. - The best forgery across every configuration I tried reached 0.616 . The best forgery stayed well below the lowest genuine score. Set the bar at 0.8, and every real chip gets in while the check turns away every forgery. Launch settings can nudge the arrival order a little, but nowhere near enough to copy another chip's. Now do it with PyTorch This is the big one. The custom kernel works, but I wanted to know whether code people already run leaks the same fingerprint. torch.index add sums values into bins using atomics, which makes it one of PyTorch's nondeterministic ops, and it shows up all over the place: embedding gradients, graph neural net aggregation and MoE routing. My first attempt fed it ordinary random data and found absolutely nothing. With lots of values per bin, the order effects average out and the sum is just noise. I was about to write the PyTorch version off right there, but on more digging it turned out the input was the problem, not the op. The trick is to make the order matter more than anything else. Give each bin only a few values, with magnitudes anywhere from 2^-30 to 2^30, so whichever value the GPU adds first decides how the rounding falls. Now the output itself carries the fingerprint: python import torch K, PER = 65536, 4 bins, values per bin g = torch.Generator device="cuda" .manual seed 20260831 same input on every chip idx = torch.arange K, device="cuda" .repeat interleave PER idx = idx torch.randperm K PER, generator=g, device="cuda" e = torch.rand K PER, generator=g, device="cuda" 60 - 30 s = torch.where torch.rand K PER, generator=g, device="cuda" 0.5, 1.0, -1.0 src = s torch.exp2 e .float magnitudes 2^-30 .. 2^30 ref = torch.zeros K, dtype=torch.float64, device="cuda" .index add 0, idx, src.double resid = torch.zeros K, dtype=torch.float64, device="cuda" for in range 400 : out = torch.zeros K, device="cuda" .index add 0, idx, src the real op resid += out.double - ref signature = torch.sign resid one bit per bin That's the whole thing: the output of an op that ships with PyTorch, and not a single custom kernel anywhere. - Chips on the same driver : 100% 9 out of 9 across three A40s . - Six A40s on mixed drivers : 78%, against 17% for guessing. The raw output picks up a little of the driver version as well as the chip. - Three RTX 4090s : 6 out of 6, with an equal error rate of 0%. - Capacity : about 217 independent bits per A40 signature and 239 on the 4090. That's enough to tell apart a very, very large fleet. Where it stops So can you work out which GPU served your chatbot reply from its logits? No. I ran a 6-layer transformer forward and backward 200 times each on three A40s, in both normal and deterministic mode. The logits and gradients were bit-identical from run to run and across all three chips. Dense matmuls and TF32 matmuls behaved the same way, and with no variation, there's nothing to fingerprint. Clock speeds and power draw under sustained load didn't separate the chips either. The leak needs atomics adding into the same place, in an order that changes the answer. Standard transformer inference avoids that, while scatter-heavy ops like index add rely on it. Witness protection for your GPU If you'd rather your GPU stayed anonymous, here's what works: - Turn on deterministic mode. torch.use deterministic algorithms True plus CUBLAS WORKSPACE CONFIG=:4096:8 made the index add output bit-identical across chips, and the PyTorch fingerprint vanished completely. It isn't free though, backward passes ran 2.5 to 3.6 times slower, and forward passes about 1.2 times. - The custom probe doesn't care. It's the tenant's own CUDA kernel, so PyTorch's flags never touch it. Stopping it means not letting people run custom kernels, hiding the %smid register, or virtualising how the GPU schedules work onto SMs, all of which are a lot heavier. The PyTorch leak is cheap to close, then, while the custom-kernel probe is very hard to close. What it's good for Cloud providers sell GPUs as interchangeable, but a tenant can tell exactly which physical chip they got. That's handy for checking you're actually on the hardware you paid for, keeping an inventory of a fleet without trusting what the host reports, or tracing which chip produced a result. The flip side is privacy: a tenant can also tell when they've landed on the same chip again. Next up, compute permitting, I want to run this on dozens of chips per model to put a proper number on collisions, try it on H100s and B200s, and go looking for more everyday ops that leak the same way, like MoE dispatch and embedding bags. Try it The repo includes every measurement from this post, so you can reproduce the numbers on a CPU without touching a GPU: git clone https://github.com/divagr18/gpu-instance-fingerprint && cd gpu-instance-fingerprint pip install -r requirements.txt python analysis/pooled analyze.py 19 A40s: 111/112 = 99.1% python analysis/analyze puf.py "a40 8chip/ .json" 8 A40s: 32/32 = 100% To fingerprint your own GPUs, you'll need CuPy for the probe, or just PyTorch for the index add version. Collect a few runs from two or more GPUs of the same model, then point the analysis at them: pip install numpy cupy-cuda12x python probes/probe puf.py 3 0 gpu a r0.json 3 epochs, run 0 python probes/probe puf.py 3 1 gpu a r1.json python analysis/analyze puf.py "path/to/ .json" python probes/passive index add.py 0 65536 passive r0.json the PyTorch version If you use it in your work, you can cite it as: @software{agrawal2026gpufingerprint, author = {Agrawal, Divyansh}, title = {Telling identical {GPUs} apart from their numerical output}, year = {2026}, url = {https://github.com/divagr18/gpu-instance-fingerprint} } Notes 1. Hohentanner et al., " Unveiling Privacy Risks in WebGPU through Hardware-based Device Fingerprinting https://doi.org/10.1145/3734477.3734700 " WiSec 2025 , and Tee and Happel, " GPU Fingerprinting for Location Verification https://arxiv.org/abs/2605.01930 " 2026 . ↩ footnote-ref-1 2. A float32 only keeps about 7 significant digits, so every addition rounds the result to the nearest number it can represent. Add a huge number and a tiny one, and the tiny one can vanish completely. Add the tiny ones together first, and they might survive. Order changes the answer. ↩ footnote-ref-2 3. An atomic add is a read-add-write that the hardware guarantees nobody else can interrupt halfway. Many threads can safely add into the same location at once, but the hardware decides the order it serves them in. ↩ footnote-ref-3