What Is Batch Invariance in LLM Inference?
Batch invariance means a request produces the same inference result when the server runs it alone, alongside other requests, at a different…
No. A seed can improve repeatability, but it does not guarantee identical large language model (LLM) output. It initializes the pseudorandom number generator used in token sampling; it does not lock the model, prompt construction, hardware, numerical calculations, or hosted infrastructure.
To replay output exactly, you usually need the same model revision, tokenizer and chat template, fully rendered prompt, decoding settings, random state, runtime, hardware behavior, batching conditions, and dynamic inputs. Hosted APIs generally offer best-effort repeatability. Self-hosted deployments can provide stronger guarantees only when engineers pin and verify the full inference environment.
An LLM generates text one token at a time. At each step, the model calculates a probability for every possible next token. The decoder then selects one token from that distribution.
With sampling enabled, decoding parameters determine how the distribution is used:
The seed initializes the pseudorandom sequence used to select tokens from the resulting distribution. If the model produces the same distribution at every step, the same seed and decoding configuration produce the same sampling choices.
That is why engineers should still set seeds for debugging, prompt comparisons, regression tests, and controlled evaluations. A seed removes avoidable sampling variance. It does not control the entire function that produces the output:
\text{output} = f(\text{weights}, \text{prompt}, \text{tokenizer}, \text{decoding}, \text{random state}, \text{runtime})
A numeric seed has meaning only within a particular implementation. Different providers and inference engines can use different pseudorandom generators, tokenizers, sampling algorithms, and seed semantics.
A seed identifies neither the model weights nor the probability distribution from which the decoder samples. Reusing seed 42
with another model does not reproduce the first model’s choices. Changing the model deployment, prompt template, tokenizer, or external input also changes the token probabilities, so the same random sequence no longer produces the same response.
Microsoft documents Azure OpenAI seeded generation as a best-effort attempt at deterministic sampling, not a guarantee. An archived OpenAI Cookbook example similarly tied repeatability to matching request parameters and system_fingerprint
, and applied its guidance to specific 2023 model versions rather than to all current models.
Treat a seed as one field in a replay record, not as a portable identifier for a response.
Temperature zero normally selects the highest-probability token instead of deliberately sampling among alternatives. This is called greedy decoding. Greedy decoding removes one source of randomness, but it does not guarantee that the model computes exactly the same probabilities on every run.
The model calculates logits, numerical scores for each possible next token. Floating-point arithmetic rounds those scores. Parallel GPU operations can also combine values in different orders. Because floating-point addition is not perfectly associative, changing the operation order, numerical precision, kernel, batch, or hardware can slightly change a logit.
If two tokens have nearly equal scores, a small numerical difference can reverse their ranking. Once one token changes, the model receives a different sequence on the next step. The remaining completion can then diverge substantially.
A seed has little direct effect during greedy decoding because the decoder selects an argmax, or highest-scoring token. The seed cannot correct a changed argmax.
Sampling randomness and numerical nondeterminism are separate problems.
A fixed seed repeats the same random draw only when the decoder uses the same probability distribution. If rounding or a different execution path changes that distribution, the same random stream can select a different token.
Temperature zero removes the random draw but still depends on stable token scores. Therefore, deterministic random-number generation and deterministic model computation require separate engineering controls.
A serving engine combines multiple requests into a batch to improve hardware utilization. Different batch contents, request order, padding, or scheduler decisions can select different GPU kernels or reduction orders. The visible prompt stays the same, but the arithmetic path changes.
The Thinking Machines Lab report on LLM nondeterminism describes this mechanism and reports an experiment in which a Qwen model generated 80 unique completions across 1,000 temperature-zero attempts under default settings. With batch-invariant kernels enabled, all 1,000 completions matched. This is engineering evidence from one implementation, not a universal benchmark, but it demonstrates why serving behavior matters.
Exact reproduction requires more than matching the visible prompt and seed. The complete execution contract includes the following variables.
Record and pin:
A provider can update a backend without changing the model name. A local deployment can also change output after an engine upgrade, a new quantization artifact, or a modified chat template.
The effective prompt includes more than the user message. Preserve:
Two requests that look identical in an application log can produce different prompts if retrieval results, timestamps, tool responses, or formatting steps differ.
Record every generation control, including:
Changing a single control changes the probability distribution or the point at which generation stops.
The serving layer can change computation through:
Hosted API callers generally cannot freeze these variables. A self-hosted deployment can control more of them, but only if its serving mode exposes the necessary settings.
For local inference, record:
Different GPU counts or precision formats can produce different arithmetic paths even with identical weights and prompts.
Two 2025 studies show why a fixed seed and temperature zero are insufficient as universal guarantees.
The ACL paper Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments tested five hosted LLM systems across eight tasks. The researchers used identical inputs,
temperature=0
, top_p=1
, and fixed seeds, with ten runs per condition. The experiments took place in February 2025.The study found accuracy variation of up to 15 percentage points across repeated runs under settings intended to be deterministic. It also reported a particularly large maximum-to-minimum gap for Mixtral-8x7B on one college-math task, where accuracy ranged from 75% to 3% across the tested runs.
Raw text agreement was often lower than parsed-answer agreement. Different explanations or formatting could still contain the same selected answer, while exact-string comparisons treated them as different. This distinction matters for downstream parsers, evaluators, and snapshot tests.
The study covered only five systems, eight tasks, and ten runs per condition, so its measured variation does not establish a failure rate for every provider or workload. It does establish that fixed seeds and zero temperature do not guarantee stable hosted output.
The NeurIPS 2025 study, Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference, varied GPU type, GPU count, batch size, and numerical precision.
For DeepSeek-R1-Distill-Qwen-7B under BF16 precision, the researchers reported up to 9 percentage points of accuracy variation across tested runtime configurations. On MATH500, more than 90% of examples diverged under BF16 across those configurations, compared with 2.2% under FP32. They also observed response-length differences of up to 9,000 tokens in tested cases.
The results do not mean that FP32 guarantees determinism in every deployment. They show that lower-precision arithmetic and runtime configuration can materially affect greedy decoding. The same seed cannot compensate for a different numerical path.
Hosted APIs are convenient but expose only part of the reproducibility contract. A provider might expose a model identifier, seed, request parameters, request ID, or backend fingerprint. The caller generally cannot freeze the provider’s GPU type, batching, scheduler, kernel libraries, numerical precision, routing, or deployment revision.
Record the provider’s available metadata, especially a system or backend fingerprint. Microsoft recommends monitoring system_fingerprint because backend changes can affect repeatability. A matching fingerprint can provide useful evidence that the backend configuration did not change, but it is not proof that every internal computation was identical.
Self-hosting provides stronger control, not automatic determinism. vLLM’s version 0.10.2 reproducibility documentation conditions reproducibility on using the same hardware and vLLM version and warns that ordinary online serving does not guarantee reproducibility. The documentation describes its newer batch-invariance feature as a beta feature designed to reduce dependence on batch size and request order, and limits it to supported NVIDIA hardware.
Choose the deployment model based on the required level of repeatability:
“Reproducible” describes several different targets. Choose the target before designing a test.
| Target | Meaning | Example test |
|---|---|---|
| Bitwise reproducibility | Intermediate computations and output bytes match | Compare recorded tensors and output hashes |
| String-level repeatability | Returned text matches exactly | Compare the two response strings |
| Answer-level repeatability | Parsed answers or structured values match | Compare labels, fields, or extracted values |
| Semantic consistency | Responses preserve the same material meaning | Use task-specific review or a validated metric |
| Statistical reproducibility | Aggregate results remain within a reported range | Compare means, variance, or confidence intervals |
A system can produce identical text and still produce an incorrect answer. It can also produce different wording while preserving the same parsed JSON value or classification label.
For JSON extraction, measure schema validity, parser success, field-level agreement, and required-value accuracy. Raw string equality is a poor primary metric if whitespace, field order, or explanatory text does not affect the downstream result.
For classification, repeat representative inputs and measure label disagreement, accuracy, and calibration. A stable label is not enough if the model is consistently wrong.
For assistants, evaluate task success, safety behavior, required facts, and important constraints across repeated runs. For benchmarks, report repeated-run distributions rather than one score when exact replay is not assured.
For audit-sensitive workflows, store the original request, response, model metadata, dynamic inputs, and evaluation result together. That record supports both exact comparison and task-level investigation.
For a hosted API, log at least:
Snapshot dynamic inputs rather than relying on the original request alone. A changed retrieval result or tool response changes the effective prompt even when the user message remains unchanged.
For self-hosted inference, pin the model and tokenizer revisions, chat template, quantization artifacts, inference-engine commit, CUDA and driver versions, kernel libraries, GPU topology, precision, and scheduler configuration. Control or record batch size, concurrency, request order, speculative decoding, caching, tensor parallelism, and prefill behavior.
Then run a representative regression suite repeatedly under production-like concurrency. Compare output hashes for string-level changes and task-level metrics for meaningful behavior changes. A seed belongs in that audit trail, but it is not evidence that the full system is replayable.
Setting a seed is useful because it reduces sampling variance and makes debugging and regression testing more controlled. It does not make LLM output inherently reproducible.
For exact text, freeze and verify the complete model-serving environment. For most production systems, test parsed answers, labels, schema validity, task success, safety behavior, or statistical performance across repeated runs instead.
No. A seed controls pseudorandom choices during sampling, but model updates, backend behavior, numerical computation, batching, and dynamic inputs remain outside its control.
They often match on a stable deployment with fixed decoding settings, but exact equality is not guaranteed. Treat repeated matching results as evidence of observed repeatability, and consult provider or serving-stack documentation for any stated guarantees.
No. Temperature zero removes deliberate sampling randomness through greedy decoding. Numerical rounding, GPU execution paths, batching, scheduling, or backend changes can still alter the highest-scoring token.
No. Providers can use different tokenizers, model distributions, pseudorandom generators, sampling implementations, and seed semantics. A seed is provider-specific rather than a portable replay key.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com