Eliminating Nondeterminism in LLM Inference with Integer-Only Arithmetic

A while back, Thinking Machines published Defeating Nondeterminism in LLM Inference, which explained why greedy decoding isn’t deterministic in production. In theory, at temperature 0 a model always picks its single most likely next token, so the same prompt gives the same answer every time. In practice, when they sent one prompt 1,000 times, they got 80 different answers.1

The problem is that floating-point addition is not associative. $(a + b) + c$ and $a + (b + c)$ can round differently. A GPU sums thousands of numbers for every token, and the order it sums them in depends on things like how many other requests are in the batch. Change the order, change the rounding, and occasionally change which token wins.