Can Provider Routing Change LLM Outputs?
Provider routing can change an LLM's output when a request reaches a different model version, fallback model, parameter configuration, precision level, inference…
Large language model (LLM) inference is the process of producing output tokens from an input prompt. It uses floating-point arithmetic for matrix multiplication, attention, normalization, and token scoring. Floating-point numbers approximate real numbers, so each operation rounds its result to a finite-precision representation. Under a fixed implementation and execution path, the arithmetic can be deterministic, but changing the operation order, precision, hardware, or kernel can change the rounded values.
That difference matters when the model chooses between nearly tied tokens. A logit is the score assigned to a possible next token. A small logit change can alter the selected token. Autoregressive generation then feeds that token back into the model, allowing a small numerical difference to produce a substantially different completion. Reproducible LLM inference therefore requires controls for both pseudorandom sampling and numerical execution.
Floating-point determinism means that a fully specified computation, including its implementation and execution conditions, produces the same rounded result each time. End-to-end determinism is stronger: it requires the entire inference path, from input preprocessing through decoding, to produce the same result.
The distinction follows from floating-point non-associativity. In finite-precision arithmetic:
(a+b)+c \ne a+(b+c)
The expressions are equal in real-number mathematics, but each intermediate addition is rounded. Parenthesization therefore changes which values are rounded and when. A parallel reduction, which combines many values through a tree of partial sums, shows how the order affects the result. Two valid reduction trees can produce different results even when every operation uses the same floating-point format and rounding mode. NVIDIA’s floating-point documentation demonstrates this behavior for single-precision arithmetic.
LLM inference contains many such reductions. Matrix multiplication computes dot products, attention sums values across tokens, normalization calculates sums or sums of squares, and distributed inference combines partial results across GPUs. If a kernel changes the accumulation order, the resulting tensors can differ.
Four reproducibility targets are useful:
| Target | Guarantee | Typical use |
|---|---|---|
| Bitwise tensor equality | ||
| Every compared tensor contains exactly the same bits. | Kernel debugging and strict regression tests | |
| Repeated-run output equality | ||
| Repeating one request produces the same tokens or final text in one pinned environment. | Local debugging and controlled evaluation | |
| Batch invariance | ||
| A request produces the same result whether it runs alone or with different requests, batch sizes, or request orders. | Production serving | |
| Cross-environment reproducibility | ||
| Results remain the same across specified GPUs, drivers, libraries, frameworks, or distributed layouts. | Scientific evaluation and audits |
These guarantees form a rough hierarchy, but they are not interchangeable. Matching final text does not prove bitwise tensor equality. Repeating a request successfully in isolation does not prove that another batch composition will produce the same result. Reproducing an aggregate score on another GPU does not prove that every token or intermediate tensor matches.
Floating-point arithmetic is therefore not inherently nondeterministic. Variation appears when an implementation changes precision, operation order, scheduling, kernel selection, workspace behavior, or hardware execution.
Run-to-run repeatability means that the same prompt, model, code, and workload produce the same result when repeated on one pinned hardware and software stack. This is the least demanding useful target.
Batch invariance means that a request’s result does not depend on its request companions. A serving engine might produce the same answer when a request runs alone but a different answer when another request changes the batch shape or scheduling path. This matters because continuous batching combines incoming requests to improve GPU utilization.
Cross-environment reproducibility covers changes such as a different GPU architecture, GPU count, CUDA toolkit, driver, BLAS library, inference engine, or tensor-parallel topology. Vendor guarantees are normally narrower than this target. For example, cuBLAS documentation scopes some bitwise guarantees to particular toolkit versions, GPU architectures, and execution conditions.
A practical decision rule is:
Order-sensitive computation appears throughout transformer inference across a range of operations.
Matrix multiplication forms each dot product by multiplying paired values and adding the products. GPU kernels divide those products among threads and partial tiles. Different tile sizes, block layouts, or Split-K
strategies change how partial sums are combined.
Attention is especially relevant. The query-key operation computes attention scores by taking dot products between a query and keys. A small difference in one score changes the subsequent softmax distribution. The softmax denominator is itself a reduction over many exponentiated scores, so its accumulation order also matters.
RMSNorm and LayerNorm reduce values across hidden dimensions. Their sums, sums of squares, and reciprocal square roots can vary with accumulation precision and order. The final output projection then reduces hidden-state products into one logit per vocabulary token.
A mixture-of-experts model routes tokens to selected subnetworks called experts. A KV cache stores prior attention keys and values during generation. An all-reduce operation combines partial tensors across devices.
Other order-sensitive paths include:
A collective operation such as all-reduce combines partial tensors from multiple devices. GPU count and communication topology affect the reduction tree, which changes the order of floating-point additions.
PyTorch’s numerical-accuracy documentation warns that mathematically equivalent batched and unbatched computations are not guaranteed to match bit for bit. The same warning applies to full-tensor and slice computations. An inference engine can therefore return a different tensor after changing batch composition even when the mathematical model is unchanged.
Dynamic batching, chunked prefill, concurrent requests, workspace availability, and stream scheduling can select different kernels or partition the same operation differently. GPU architecture, library version, and tensor-parallel topology also vary the execution path.
This explains why an isolated request can be repeatable while a production request is batch-sensitive. The arithmetic remains valid in both cases, but the implementation performs reductions in a different order.
Precision affects how much rounding occurs, but precision alone does not determine end-to-end determinism.
Accumulation also affects numerical behavior. A model can store weights and activations in FP16 or BF16 while accumulating products in FP32. A kernel can also use higher precision for selected operations such as normalization or logits. These hybrid approaches reduce memory use while limiting numerical error.
A NeurIPS 2025 study found BF16 less stable than FP16 and FP32 in its tested LLM configurations. For DeepSeek-R1-Distill-Qwen-7B under the tested BF16 configuration changes, the study reported up to 9% accuracy variation and up to 9,000 tokens of response-length difference when GPU type, GPU count, and evaluation batch size changed.
Those results demonstrate configuration sensitivity, not inevitable divergence on every repeated call in one unchanged environment. FP32 reduced instability in the tested configurations, with no guarantee of determinism across library versions, GPU architectures, kernels, or schedules.
Precision error and execution variation are different problems:
The causal chain has four stages:
Greedy decoding selects the token with the highest logit:
\operatorname{argmax}_v \; \text{logit}(v)
Suppose two candidate tokens have these logits:
| Token | Run A | Run B |
|---|---|---|
A |
||
| slightly higher | slightly lower | |
B |
||
| slightly lower | slightly higher |
Run A selects A
, while Run B selects B
. The numerical changes are tiny, but the output decision is discrete.
Temperature zero removes sampling variation by selecting the highest-scoring token instead of drawing from a probability distribution. It does not make different logits identical. If numerical variation swaps the top two logits, temperature-zero decoding still produces different tokens.
After the first different token, the next model call receives a different prefix. Every later attention calculation and token score now depends on that changed prefix. The divergence can affect reasoning steps, response length, tool calls, generated code, and the final answer.
The most useful diagnostics are not limited to final-string comparisons. Record:
The first divergence location identifies where a continuous numerical difference became a discrete decoding difference.
The strongest directly relevant peer-reviewed evidence is the NeurIPS 2025 study described above. It tested precision, GPU type, GPU count, and batch size across several LLMs. Its results support the claim that LLM evaluation outcomes can depend on numerical format and runtime configuration. They do not establish that all LLMs diverge under all repeated-run conditions.
A separate Thinking Machines Lab report provides serving-time evidence. It reported 80 unique completions across 1,000 temperature-zero trials using Qwen/Qwen3-235B-A22B-Instruct-2507
; the most common completion appeared 78 times. This is evidence about the tested serving configuration, not a universal measurement of LLM behavior. The report is also not peer reviewed.
The two findings address different scopes:
A 2024 IEEE/SC reduction study also shows why performance claims need qualification. In a parallel-sum microbenchmark, one deterministic reduction strategy cost less than 0.2% on an NVIDIA V100 but 7.8% on an NVIDIA GH200. This is not an end-to-end LLM benchmark, but it demonstrates that deterministic reductions have hardware-dependent costs rather than one fixed penalty. The study supports benchmarking the target stack instead of assuming that deterministic execution is either free or always impractical.
Apparently conflicting explanations become consistent when separated into three layers:
Dynamic batching alone does not create numerical error. It changes the computation path that interacts with order-sensitive arithmetic. Under fixed conditions, a GPU execution path can be deterministic. A different path can still produce a different, valid floating-point result.
A seed controls pseudorandom-number generation. It does not specify kernel selection, reduction order, workspace allocation, stream scheduling, GPU architecture, or collective communication order.
PyTorch’s reproducibility documentation states that identical results are not guaranteed across releases, commits, platforms, or CPU and GPU executions, even when the same seeds are used. Framework deterministic flags help only for supported operations. They can select slower algorithms, increase memory use, or fail to cover an operation used by the model.
TensorFlow documents a similar boundary. Its deterministic-operation mode addresses variation caused by asynchronous GPU accumulation for eligible operations, but it warns that deterministic alternatives can reduce performance and that some operations lack deterministic implementations.
NVIDIA library guarantees are also scoped. cuBLAS documents conditions under which a toolkit version can provide repeatable results on GPUs with the same architecture and configuration, while excluding cases such as toolkit changes and certain multi-stream or workspace behaviors. cuDNN documentation similarly distinguishes repeatability on the same architecture from reproducibility across architectures.
Treat determinism as an execution contract, not a switch. The contract should name:
Deterministic inference can reduce throughput, increase latency, consume more memory, restrict batching, or disable fast kernels. The cost depends on the model, sequence lengths, precision, concurrency, GPU architecture, and serving engine.
The cost is not always large. The reduction benchmark above found a small overhead on one GPU and a larger overhead on another. That result does not predict LLM serving performance, but it shows why measurements must use the actual deployment.
Benchmark at least these conditions:
vLLM’s batch-invariance documentation describes a beta mode designed to make outputs independent of batch size and request order. It uses deterministic kernels and disables some optimizations, so its documented model and hardware limitations, as well as its performance effects, must be tested rather than assumed away.
When exact regeneration costs too much, use a weaker contract. Tolerance-based logit checks, probability-margin checks, semantic evaluation, or cached approved outputs can provide more operational value than universal bitwise equality.
Start by choosing the guarantee. Do not configure controls before deciding whether the requirement is exact replay, batch-invariant serving, or cross-environment scientific reproducibility.
Then pin the complete execution input:
Enable documented deterministic algorithms where supported. Use stable workspace and stream settings when the relevant library documents them. Prefer batch-invariant kernels or controlled scheduling when serving reproducibility matters.
Test more than final text. Compare logits, token probabilities, top-1/top-2 margins, first divergence position, batch composition, request order, and isolated versus concurrent execution. A final answer can match even when intermediate tensors differ, and a final answer can diverge because of one near-tied token despite otherwise small numerical differences.
Store reproducibility metadata with the experiment or evaluation artifact, not only in a temporary debugging log. A useful manifest includes:
For strict audit replay, cache approved responses and their token sequences. A cache guarantees replay for known inputs more reliably than attempting to regenerate them after the serving stack changes.
Floating-point determinism affects LLM reproducibility because order-sensitive arithmetic creates small numerical differences. Runtime configuration triggers those differences, and autoregressive token selection amplifies them into different text. Seeds and temperature zero address sampling, but they do not guarantee identical numerical execution.
Reliable reproducibility requires a stated equality target, a pinned model and runtime environment, controlled batching and kernels, suitable precision, and diagnostics that capture logits and the first divergent token. When strict regeneration is too costly, tolerance-based evaluation or cached outputs provides a narrower, more practical contract.
No. Selecting the highest-scoring token at temperature zero removes sampling variation, but it does not guarantee identical logits across different kernels, batches, GPUs, or floating-point paths. A small numerical shift can change the selected token when two candidates are nearly tied.
A seed controls pseudorandom-number generation, not the entire execution graph. It does not force the same reduction order, kernel, workspace, stream schedule, precision, hardware, or distributed communication path.
No. A fully specified operation is deterministic for a chosen format, rounding mode, and implementation. Different valid reduction orders produce different rounded results because floating-point addition is not associative.
Batch composition can change kernel selection, work partitioning, scheduling, and reduction order. A batch-invariant serving path must preserve the request’s numerical procedure even when other requests join, leave, or change its batch.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com