temperature=0 Isn't Deterministic: 17 of 5,000 Greedy Tokens Flipped

I had a regression test that sent the same prompt to an LLM endpoint with temperature=0 and diffed the output against a saved golden file. It passed on Monday. It failed on Tuesday. Same prompt, same model, same parameters. The two outputs matched word for word for about 140 tokens, then one said "however" where the other said "but", and from there they were two different essays.

I assumed the provider had swapped the model. They hadn't. The actual reason: temperature=0 isn't deterministic, and it's not a bug in anyone's sampler. It's floating-point arithmetic plus the fact that your request shares a GPU with strangers.

temperature=0 means greedy decoding: pick the argmax token every step. Picking an argmax is deterministic, but computing the logits it's picked from is not.(1024 + 0.5) + 0.5 gives 1024, while 1024 + (0.5 + 0.5) gives 1025. Because the logits shift slightly from run to run, and greedy decoding is only deterministic when its inputs are. Temperature scales the logits before sampling. At zero, every serious library swaps in a plain argmax. That part is fine.

The problem is upstream. Every logit is a dot product: a hidden vector of a few thousand numbers multiplied element-wise by a row of the output projection and summed. Summing a few thousand floats in a different order gives a slightly different answer. When the top two candidates are close, "slightly different" is enough to change which one wins.

Here's the smallest demo of that I know:

import numpy as np

a, b, c = np.float16(1024.0), np.float16(0.5), np.float16(0.5)
print((a + b) + c)   # 1024.0
print(a + (b + c))   # 1025.0

At 1024, float16 can only represent steps of 1.0. Adding 0.5 rounds back to 1024 (round half to even), twice. Adding 1.0 at once survives. Same three numbers, different order, different result. Scale that up to 4,096-element dot products across a 100K-token vocabulary and you get logits that move in the third decimal place depending on how the sum was grouped.

The kernel's tiling strategy changes it, and the strategy depends on the shape of the work. A matrix multiply doesn't add 4,096 products in one line. It splits the reduction dimension into tiles, computes partial sums in parallel, then combines them. Libraries like cuBLAS pick different tile sizes and "split-K" strategies depending on matrix dimensions, because the fastest kernel for a batch of 1 isn't the fastest for a batch of 64.

So the summation order for your request depends on:

Each forward pass is deterministic in isolation. Run the same batch on the same machine twice and you'll usually get identical bits. What isn't fixed is which batch your request lands in. That's the key insight, and it's why "the GPU is nondeterministic" is the wrong mental model. The real issue is batch invariance: your output depends on who you were batched with.

I wanted a number instead of a vibe, so I wrote a toy. It's not a real model. It's random float16 weights for an 8,000-token "vocabulary" with a 4,096-dimension hidden state, and the only variable is how the dot product is chunked: partial sums of 256 elements versus 1,024 elements, each accumulated into a float16 running total.

import numpy as np
rng = np.random.default_rng(0)
V, D, N = 8000, 4096, 5000
W = (rng.standard_normal((V, D)) / np.sqrt(D)).astype(np.float16)
H = rng.standard_normal((N, D)).astype(np.float16)

def logits(h, chunk):
    acc = np.zeros(V, dtype=np.float16)
    for s in range(0, D, chunk):
        part = W[:, s:s+chunk].astype(np.float32) @ h[s:s+chunk].astype(np.float32)
        acc = (acc + part.astype(np.float16)).astype(np.float16)
    return acc

flips = sum(logits(h, 256).argmax() != logits(h, 1024).argmax() for h in H)
print(flips, "of", N)

Results from that run:

0.34% sounds harmless until you remember generation is sequential. If each step independently had a 0.34% chance of flipping, a 500-token answer would have 1 - (1 - 0.0034)^500 ≈ 0.82, so an 82% chance of at least one divergence.

Two big caveats. Positions aren't independent in a real model, and real logit distributions aren't Gaussian. Real models often have genuine near-ties ("but" versus "however", a comma versus a period), which pushes the flip rate up at exactly the positions where language is ambiguous. Confident positions, like the closing bracket in code, almost never flip. Treat my 82% as an illustration of compounding, not a measurement of any production model.

Because every later token is conditioned on every earlier one. Once position 140 says "however" instead of "but", position 141 sees a different prefix, gets different hidden states, and the logits are no longer the same distribution at all. It's not a small error that persists. It's a fork.

That's why nondeterministic outputs look the way mine did: identical for a stretch, then fully different. If you ever diff two temperature=0 outputs and see a shared prefix that suddenly splits, you're looking at a single argmax flip, not a model swap.

No. A seed controls the random number generator used for sampling. Greedy decoding never touches the random number generator, so a seed has nothing to fix. The variance comes from arithmetic, not randomness.

The same goes for torch.use_deterministic_algorithms(True) and CUBLAS_WORKSPACE_CONFIG=:4096:8. Those make a given operation on a given shape reproducible. They don't make a batch of 1 and a batch of 37 produce identical bits for your row. You still need the shapes to match, or kernels that don't care about the batch dimension.

It depends on whether you own the inference server.

If you run the model yourself:

If you call a hosted API:

temperature=0 isn't deterministic because greedy decoding takes the argmax of logits that are computed with floating-point sums, and the order of those sums depends on how the GPU kernel tiles the work, which depends on batch size, which depends on other traffic hitting the same server. Floating-point addition isn't associative, so different groupings produce slightly different logits, and when the top two tokens are nearly tied the winner flips. One flip forks every token after it. A seed doesn't help, since greedy decoding uses no randomness. Batch-invariant kernels fix it if you own the server. If you don't, cache the outputs you need to replay and test for meaning rather than exact strings.

Written by the developer behind Preterview, an interview prep platform.