Our CI had one test that failed roughly once a week. Same prompt, same model snapshot, temperature=0, seed pinned, snapshot assertion on the output string. Nothing in the diff touched it.
I did what every engineer does with a weekly flake: reran it, watched it go green, and blamed the network.
Then I got annoyed enough to loop the exact same request 500 times and diff the results. Twelve of them came back different. Not "slightly reworded" different in the wishy-washy sense — one of them classified a refund ticket as billing instead of fraud, which is a real behavior change from a byte-identical request.
Temperature 0 isn't deterministic. It never was. And the reason is not the model being creative behind your back.
temperature=0 makes sampling greedy (always pick the top token), but it does Because greedy decoding is deterministic given identical logits, and you never get identical logits from a shared inference server.
Walk the pipeline. Your prompt becomes a matmul-heavy forward pass that ends in a vector of scores over the vocabulary. temperature=0 collapses the sampler to argmax. That part is genuinely deterministic — argmax over the same floats returns the same index every time.
The floats are the problem.
Floating point addition is not associative. (a + b) + c and a + (b + c) can differ in the last bits. On a CPU running one thread you would never notice, because the order never changes. On a GPU, a reduction (summing across a hidden dimension, softmax denominators, RMSNorm) gets split across blocks and the results combined. The way it splits depends on the tensor shape the kernel was handed.
And the shape depends on the batch.
Batch invariance means a single request produces the same output no matter what else is batched alongside it. Most production inference kernels do not have this property, and it's a performance decision, not a bug.
Serving stacks batch aggressively. Your request at 3pm on a Tuesday lands in a batch of 48 other requests. The same request at 4am lands in a batch of 3. Different batch size, different tiling strategy, different split-K choice inside the matmul, different reduction order, different last-bit rounding on your logits.
Nobody is randomizing anything. Every individual run is fully deterministic given its batch. You just don't control your batch, and you can't see it.
A few other things stack on top of this on real endpoints:
The seed parameter on the major APIs is documented as best effort for exactly this reason. It pins the sampler's RNG. It does not pin the arithmetic.
Because decoding is autoregressive, so a single token swap changes the input to every token after it.
Most positions are not close calls. When the model writes "the capital of France is Paris", the gap between the top token and the runner-up is enormous, and a 1e-7 perturbation cannot touch it. Those tokens are effectively locked.
But a small share of positions are near-ties: "however" vs "but", { vs [", fraud vs billing on an ambiguous ticket. At those positions the top-2 gap is smaller than your numeric noise, and the argmax genuinely coin-flips.
Then the divergence compounds. Once the model has committed to "however", it is conditioning every future token on "however". You don't get a one-word diff. You get a different second half.
This is also why the flakes cluster on your hardest inputs. Ambiguous, low-confidence prompts are exactly where near-ties live. Your test suite's easy cases stay green forever and your nastiest edge case fails once a week, which reads like a haunting.
Mostly you don't, on a hosted API. You make your tests robust and reserve bitwise determinism for cases where you own the whole stack.
What actually worked for us:
parsed.label in ALLOWED and parsed.label == "fraud", not the surrounding prose. system_fingerprint or equivalent) so you can tell "backend changed" apart from "we hit a near-tie."
If you truly need bitwise reproducibility, run locally: batch size 1, fixed library and driver versions, deterministic kernel flags, one GPU model. That is achievable, and it is still only reproducible against that box. Batch-invariant kernels exist and are being built into serving stacks, but they cost throughput, so you should assume you don't have them unless a vendor says so explicitly.
Say it plainly: temperature=0 controls the sampler, not the arithmetic, and the arithmetic depends on server load.
The common misread is that the model is "still being creative" and someone proposes cranking temperature to -0 or adding top_p=0 (which does nothing here) or retrying until the output matches. I've watched a team add a retry loop that reran the model until it produced the expected string. That is not determinism. That is rejection sampling with extra steps and a much larger bill.
The honest framing: an LLM call is a statistical dependency, not a pure function. You wouldn't snapshot-test a call to a third-party ranking service and expect byte equality across a year. Same posture here.
No. Temperature 0 isn't deterministic in practice, because it only makes token selection greedy while leaving the underlying computation free to vary. Floating point addition is non-associative, GPU kernels choose their reduction order based on batch shape, and batch shape on a hosted API depends on concurrent traffic you don't control. Identical prompts therefore produce slightly different logits, and at the small fraction of positions where the top two tokens are nearly tied, argmax flips — after which autoregressive decoding amplifies that single token into a different answer. You can get bitwise reproducibility on your own hardware at batch size 1 with pinned versions, or with genuinely batch-invariant kernels; on a shared endpoint you should design for a measured flip rate instead.
If your LLM test suite has one test that fails once a week, it isn't cursed. It's just standing on a near-tie.