{"slug": "how-floating-point-determinism-affects-llm-reproducibility", "title": "How Floating-Point Determinism Affects LLM Reproducibility", "summary": "Floating-point arithmetic non-determinism can cause LLM inference to produce different outputs across runs, hardware, or provider routing, according to an analysis of numerical execution in large language models. The article distinguishes four reproducibility targets—bitwise tensor equality, repeated-run output equality, batch invariance, and cross-environment reproducibility—and notes that small logit changes from rounding can alter token selection and cascade into substantially different completions.", "body_md": "### Can Provider Routing Change LLM Outputs?\n\nProvider routing can change an LLM's output when a request reaches a different model version, fallback model, parameter configuration, precision level, inference…\n\nLarge language model (LLM) inference is the process of producing output tokens from an input prompt. It uses floating-point arithmetic for [matrix multiplication, attention, normalization, and token scoring](https://proceedings.neurips.cc/paper_files/paper/2025/file/f80094a824ba5912d4a2de169c404a40-Paper-Conference.pdf). [Floating-point numbers](https://docs.nvidia.com/cuda/archive/10.0/floating-point/index.html) approximate real numbers, so each operation rounds its result to a finite-precision representation. Under a fixed implementation and execution path, the arithmetic can be deterministic, but changing the operation order, precision, hardware, or kernel can change the rounded values.\n\nThat difference matters when the model chooses between nearly tied tokens. A logit is the score assigned to a possible next token. A [small logit change](https://proceedings.neurips.cc/paper_files/paper/2025/file/f80094a824ba5912d4a2de169c404a40-Paper-Conference.pdf) can alter the selected token. Autoregressive generation then feeds that token back into the model, allowing a small numerical difference to produce a substantially different completion. Reproducible LLM inference therefore requires controls for both pseudorandom sampling and numerical execution.\n\nFloating-point determinism means that a fully specified computation, including its implementation and execution conditions, produces the same rounded result each time. End-to-end determinism is stronger: it requires the entire inference path, from input preprocessing through decoding, to produce the same result.\n\nThe distinction follows from floating-point non-associativity. In finite-precision arithmetic:\n\n```\n(a+b)+c \\ne a+(b+c)\n```\n\nThe expressions are equal in real-number mathematics, but each intermediate addition is rounded. Parenthesization therefore changes which values are rounded and when. [A parallel reduction](https://www.sciencedirect.com/science/article/pii/S0167819115001155), which combines many values through a tree of partial sums, shows how the order affects the result. Two valid reduction trees can produce different results even when every operation uses the same floating-point format and rounding mode. [NVIDIA’s floating-point documentation](https://docs.nvidia.com/cuda/archive/10.0/floating-point/index.html) demonstrates this behavior for single-precision arithmetic.\n\nLLM inference contains many such reductions. Matrix multiplication computes dot products, attention sums values across tokens, normalization calculates sums or sums of squares, and distributed inference combines partial results across GPUs. If a kernel changes the accumulation order, the resulting tensors can differ.\n\nFour reproducibility targets are useful:\n\n| Target | Guarantee | Typical use |\n|---|---|---|\nBitwise tensor equality |\nEvery compared tensor contains exactly the same bits. | Kernel debugging and strict regression tests |\nRepeated-run output equality |\nRepeating one request produces the same tokens or final text in one pinned environment. | Local debugging and controlled evaluation |\nBatch invariance |\nA request produces the same result whether it runs alone or with different requests, batch sizes, or request orders. | Production serving |\nCross-environment reproducibility |\nResults remain the same across specified GPUs, drivers, libraries, frameworks, or distributed layouts. | Scientific evaluation and audits |\n\nThese guarantees form a rough hierarchy, but they are not interchangeable. Matching final text does not prove bitwise tensor equality. Repeating a request successfully in isolation does not prove that another batch composition will produce the same result. Reproducing an aggregate score on another GPU does not prove that every token or intermediate tensor matches.\n\nFloating-point arithmetic is therefore not inherently nondeterministic. Variation appears when an implementation changes precision, operation order, scheduling, kernel selection, workspace behavior, or hardware execution.\n\n**Run-to-run repeatability** means that the same prompt, model, code, and workload produce the same result when repeated on one pinned hardware and software stack. This is the least demanding useful target.\n\n[ Batch invariance](https://docs.vllm.ai/en/stable/features/batch_invariance) means that a request’s result does not depend on its request companions. A serving engine might produce the same answer when a request runs alone but a different answer when another request changes the batch shape or scheduling path. This matters because continuous batching combines incoming requests to improve GPU utilization.\n\n**Cross-environment reproducibility** covers changes such as a different GPU architecture, GPU count, CUDA toolkit, driver, BLAS library, inference engine, or tensor-parallel topology. Vendor guarantees are normally narrower than this target. For example, [cuBLAS documentation](https://docs.nvidia.com/cuda/cublas/index.html?highlight=CUBLAS_EMULATION_STRATEGY) scopes some bitwise guarantees to particular toolkit versions, GPU architectures, and execution conditions.\n\nA practical decision rule is:\n\nOrder-sensitive computation appears throughout transformer inference across a range of operations.\n\nMatrix multiplication forms each dot product by multiplying paired values and adding the products. GPU kernels divide those products among threads and partial tiles. Different tile sizes, block layouts, or `Split-K`\n\nstrategies change how partial sums are combined.\n\nAttention is especially relevant. The query-key operation computes attention scores by taking dot products between a query and keys. A small difference in one score changes the subsequent softmax distribution. The softmax denominator is itself a reduction over many exponentiated scores, so its accumulation order also matters.\n\nRMSNorm and LayerNorm reduce values across hidden dimensions. Their sums, sums of squares, and reciprocal square roots can vary with accumulation precision and order. The final output projection then reduces hidden-state products into one logit per vocabulary token.\n\nA mixture-of-experts model routes tokens to selected subnetworks called experts. A KV cache stores prior attention keys and values during generation. An all-reduce operation combines partial tensors across devices.\n\nOther order-sensitive paths include:\n\nA collective operation such as all-reduce combines partial tensors from multiple devices. GPU count and communication topology affect the reduction tree, which changes the order of floating-point additions.\n\n[PyTorch’s numerical-accuracy documentation](https://docs.pytorch.org/docs/stable/notes/numerical_accuracy.html) warns that mathematically equivalent batched and unbatched computations are not guaranteed to match bit for bit. The same warning applies to full-tensor and slice computations. An inference engine can therefore return a different tensor after changing batch composition even when the mathematical model is unchanged.\n\nDynamic batching, chunked prefill, concurrent requests, workspace availability, and stream scheduling can select different kernels or partition the same operation differently. GPU architecture, library version, and tensor-parallel topology also vary the execution path.\n\nThis explains why an isolated request can be repeatable while a production request is batch-sensitive. The arithmetic remains valid in both cases, but the implementation performs reductions in a different order.\n\nPrecision affects how much rounding occurs, but precision alone does not determine end-to-end determinism.\n\nAccumulation also affects numerical behavior. A model can store weights and activations in FP16 or BF16 while accumulating products in FP32. A kernel can also use higher precision for selected operations such as normalization or logits. These hybrid approaches reduce memory use while limiting numerical error.\n\nA [NeurIPS 2025 study](https://proceedings.neurips.cc/paper_files/paper/2025/file/f80094a824ba5912d4a2de169c404a40-Paper-Conference.pdf) found BF16 less stable than FP16 and FP32 in its tested LLM configurations. For DeepSeek-R1-Distill-Qwen-7B under the tested BF16 configuration changes, the study reported up to 9% accuracy variation and up to 9,000 tokens of response-length difference when GPU type, GPU count, and evaluation batch size changed.\n\nThose results demonstrate configuration sensitivity, not inevitable divergence on every repeated call in one unchanged environment. FP32 reduced instability in the tested configurations, with no guarantee of determinism across library versions, GPU architectures, kernels, or schedules.\n\nPrecision error and execution variation are different problems:\n\nThe causal chain has four stages:\n\nGreedy decoding selects the token with the highest logit:\n\n```\n\\operatorname{argmax}_v \\; \\text{logit}(v)\n```\n\nSuppose two candidate tokens have these logits:\n\n| Token | Run A | Run B |\n|---|---|---|\n`A` |\nslightly higher | slightly lower |\n`B` |\nslightly lower | slightly higher |\n\nRun A selects `A`\n\n, while Run B selects `B`\n\n. The numerical changes are tiny, but the output decision is discrete.\n\n[Temperature zero removes sampling variation](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) by selecting the highest-scoring token instead of drawing from a probability distribution. It does not make different logits identical. If numerical variation swaps the top two logits, temperature-zero decoding still produces different tokens.\n\nAfter the first different token, the next model call receives a different prefix. Every later attention calculation and token score now depends on that changed prefix. The divergence can affect reasoning steps, response length, tool calls, generated code, and the final answer.\n\nThe most useful diagnostics are not limited to final-string comparisons. Record:\n\nThe first divergence location identifies where a continuous numerical difference became a discrete decoding difference.\n\nThe strongest directly relevant peer-reviewed evidence is the NeurIPS 2025 study described above. It tested precision, GPU type, GPU count, and batch size across several LLMs. Its results support the claim that LLM evaluation outcomes can depend on numerical format and runtime configuration. They do not establish that all LLMs diverge under all repeated-run conditions.\n\nA separate [Thinking Machines Lab report](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) provides serving-time evidence. It reported 80 unique completions across 1,000 temperature-zero trials using `Qwen/Qwen3-235B-A22B-Instruct-2507`\n\n; the most common completion appeared 78 times. This is evidence about the tested serving configuration, not a universal measurement of LLM behavior. The report is also not peer reviewed.\n\nThe two findings address different scopes:\n\nA 2024 IEEE/SC reduction study also shows why performance claims need qualification. In a parallel-sum microbenchmark, one deterministic reduction strategy cost less than 0.2% on an NVIDIA V100 but 7.8% on an NVIDIA GH200. This is not an end-to-end LLM benchmark, but it demonstrates that deterministic reductions have hardware-dependent costs rather than one fixed penalty. [The study](https://conferences.computer.org/sc-wpub/pdfs/SC-W2024-6oZmigAQfgJ1GhPL0yE3pS/555400a170/555400a170.pdf) supports benchmarking the target stack instead of assuming that deterministic execution is either free or always impractical.\n\nApparently conflicting explanations become consistent when separated into three layers:\n\nDynamic batching alone does not create numerical error. It changes the computation path that interacts with order-sensitive arithmetic. Under fixed conditions, a GPU execution path can be deterministic. A different path can still produce a different, valid floating-point result.\n\nA seed controls pseudorandom-number generation. It does not specify kernel selection, reduction order, workspace allocation, stream scheduling, GPU architecture, or collective communication order.\n\n[PyTorch’s reproducibility documentation](https://docs.pytorch.org/docs/stable/notes/randomness) states that identical results are not guaranteed across releases, commits, platforms, or CPU and GPU executions, even when the same seeds are used. Framework deterministic flags help only for supported operations. They can select slower algorithms, increase memory use, or fail to cover an operation used by the model.\n\nTensorFlow documents a similar boundary. Its [deterministic-operation mode](https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism) addresses variation caused by asynchronous GPU accumulation for eligible operations, but it warns that deterministic alternatives can reduce performance and that some operations lack deterministic implementations.\n\nNVIDIA library guarantees are also scoped. cuBLAS documents conditions under which a toolkit version can provide repeatable results on GPUs with the same architecture and configuration, while excluding cases such as toolkit changes and certain multi-stream or workspace behaviors. [cuDNN documentation](https://docs.nvidia.com/deeplearning/cudnn/backend/latest/developer/misc.html) similarly distinguishes repeatability on the same architecture from reproducibility across architectures.\n\nTreat determinism as an execution contract, not a switch. The contract should name:\n\nDeterministic inference can reduce throughput, increase latency, consume more memory, restrict batching, or disable fast kernels. The cost depends on the model, sequence lengths, precision, concurrency, GPU architecture, and serving engine.\n\nThe cost is not always large. The reduction benchmark above found a small overhead on one GPU and a larger overhead on another. That result does not predict LLM serving performance, but it shows why measurements must use the actual deployment.\n\nBenchmark at least these conditions:\n\n[vLLM’s batch-invariance documentation](https://docs.vllm.ai/en/stable/features/batch_invariance) describes a beta mode designed to make outputs independent of batch size and request order. It uses deterministic kernels and disables some optimizations, so its documented model and hardware limitations, as well as its performance effects, must be tested rather than assumed away.\n\nWhen exact regeneration costs too much, use a weaker contract. Tolerance-based logit checks, probability-margin checks, semantic evaluation, or cached approved outputs can provide more operational value than universal bitwise equality.\n\nStart by choosing the guarantee. Do not configure controls before deciding whether the requirement is exact replay, batch-invariant serving, or cross-environment scientific reproducibility.\n\nThen pin the complete execution input:\n\nEnable documented deterministic algorithms where supported. Use stable workspace and stream settings when the relevant library documents them. Prefer batch-invariant kernels or controlled scheduling when serving reproducibility matters.\n\nTest more than final text. Compare logits, token probabilities, top-1/top-2 margins, first divergence position, batch composition, request order, and isolated versus concurrent execution. A final answer can match even when intermediate tensors differ, and a final answer can diverge because of one near-tied token despite otherwise small numerical differences.\n\nStore reproducibility metadata with the experiment or evaluation artifact, not only in a temporary debugging log. A useful manifest includes:\n\nFor strict audit replay, cache approved responses and their token sequences. A cache guarantees replay for known inputs more reliably than attempting to regenerate them after the serving stack changes.\n\nFloating-point determinism affects LLM reproducibility because order-sensitive arithmetic creates small numerical differences. Runtime configuration triggers those differences, and autoregressive token selection amplifies them into different text. Seeds and temperature zero address sampling, but they do not guarantee identical numerical execution.\n\nReliable reproducibility requires a stated equality target, a pinned model and runtime environment, controlled batching and kernels, suitable precision, and diagnostics that capture logits and the first divergent token. When strict regeneration is too costly, tolerance-based evaluation or cached outputs provides a narrower, more practical contract.\n\nNo. Selecting the highest-scoring token at temperature zero removes sampling variation, but it does not guarantee identical logits across different kernels, batches, GPUs, or floating-point paths. A small numerical shift can change the selected token when two candidates are nearly tied.\n\nA seed controls pseudorandom-number generation, not the entire execution graph. It does not force the same reduction order, kernel, workspace, stream schedule, precision, hardware, or distributed communication path.\n\nNo. A fully specified operation is deterministic for a chosen format, rounding mode, and implementation. Different valid reduction orders produce different rounded results because floating-point addition is not associative.\n\nBatch composition can change kernel selection, work partitioning, scheduling, and reduction order. A batch-invariant serving path must preserve the request’s numerical procedure even when other requests join, leave, or change its batch.\n\nGive Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.\n\nTake a look at vroni.com", "url": "https://wpnews.pro/news/how-floating-point-determinism-affects-llm-reproducibility", "canonical_source": "https://www.vincentschmalbach.com/floating-point-determinism-llm-reproducibility/", "published_at": "2026-08-11 11:25:55+00:00", "updated_at": "2026-08-11 11:52:39.616118+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure"], "entities": ["NVIDIA"], "alternates": {"html": "https://wpnews.pro/news/how-floating-point-determinism-affects-llm-reproducibility", "markdown": "https://wpnews.pro/news/how-floating-point-determinism-affects-llm-reproducibility.md", "text": "https://wpnews.pro/news/how-floating-point-determinism-affects-llm-reproducibility.txt", "jsonld": "https://wpnews.pro/news/how-floating-point-determinism-affects-llm-reproducibility.jsonld"}}