Eliminating Nondeterminism in LLM Inference with Integer-Only Arithmetic

Thinking Machines' analysis of nondeterminism in LLM inference found that sending one prompt 1,000 times at temperature 0 produced 80 different answers, because floating-point addition is not associative and GPU summation order shifts with batch composition. The rounding differences can occasionally change which token wins, undermining the assumption that greedy decoding returns the same output every time.

A while back, Thinking Machines published Defeating Nondeterminism in LLM Inference https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ , which explained why greedy decoding isn’t deterministic in production. In theory, at temperature 0 a model always picks its single most likely next token, so the same prompt gives the same answer every time. In practice, when they sent one prompt 1,000 times, they got 80 different answers. 1 fn:1 The problem is that floating-point addition is not associative . $ a + b + c$ and $a + b + c $ can round differently. A GPU sums thousands of numbers for every token, and the order it sums them in depends on things like how many other requests are in the batch. Change the order, change the rounding, and occasionally change which token wins.