# What Is Batch Invariance in LLM Inference?

> Source: <https://www.vincentschmalbach.com/batch-invariance-llm-inference/>
> Published: 2026-08-11 09:13:46+00:00

### How Does Mixture-of-Experts Routing Affect LLM Repeatability?

Mixture-of-Experts (MoE) models split a large feed-forward network into smaller experts. A router selects only a few experts per token, improving efficiency…

Batch invariance means a request produces the same inference result when the server runs it alone, alongside other requests, at a different batch position, or under another supported batching schedule. It matters because large language model (LLM) servers group requests dynamically to use GPUs efficiently, and that grouping can change the floating-point calculations used to produce the next token.

With a declared hardware and software environment, batch-invariant inference removes batch size, batch composition, request order, and scheduling as sources of output drift. It does not guarantee identical results across every GPU, CUDA version, precision, model revision, or sampling configuration.

Here, `x`

is one logical request, including its prompt, model, decoding settings, and request-local state. A batch `B`

contains `x`

, and `E`

is a fixed execution environment.

A system is batch-invariant for `x`

when:

```
F(x; B_1, E) = F(x; B_2, E)
```

for supported batches `B_1`

and `B_2`

that contain the same request. The batches may differ in size, companion requests, request order, or scheduling.

At minimum, the controlled environment must include:

A **strong** form of batch invariance preserves intermediate values or logits: the numerical outputs before token selection remain identical. An application-level form preserves the generated token IDs or final text. The second guarantee is weaker because different intermediate values can still produce the same selected tokens.

Batch invariance concerns numerical behavior. It does not mean that batch size leaves throughput, latency, memory use, or GPU utilization unchanged. It also does not promise equal treatment between requests, which is a scheduling and fairness question.

The [vLLM batch-invariance documentation](https://docs.vllm.ai/en/v0.17.1/features/batch_invariance) describes the feature operationally: a request should produce outputs independent of batch size and request order under supported conditions.

Batch invariance is narrower than full determinism.

A random seed controls the pseudorandom sequence used during sampling. It does not force the server to calculate identical logits if a different batch shape changes the arithmetic path before sampling. Reproducible sampled generation therefore requires both a controlled random-state policy and stable numerical execution.

Batch effects do not require other requests to share prompt data with the target request. Their presence can change the total tensor shape, kernel selection, reduction partitioning, or execution schedule. The target request still uses its own logical input, but the GPU may perform the arithmetic differently.

Batch invariance is also narrower than cross-platform reproducibility. A guarantee that covers batch composition on one GPU configuration does not automatically cover:

The term is unrelated to Batch Normalization, a training-time neural-network layer. Batch invariance concerns how inference kernels calculate a request when the serving batch changes.

Dynamic batching is the starting point. In continuous batching, requests join or leave decoding steps as other requests finish. The active batch can therefore change while a request is generating tokens.

A change in batch shape can cause the runtime to select different:

The model weights and prompt remain unchanged, but the numerical execution path does not.

The underlying issue is finite-precision arithmetic. Floating-point addition is not associative after rounding:

```
(a+b)+c \ne a+(b+c)
```

Adding the same values in different orders during a reduction can therefore produce slightly different results. Normalization, matrix multiplication, attention, softmax-related calculations, and distributed communication all contain reductions that are sensitive to this effect.

A simple mental model is:

Batching changes the route used to perform the arithmetic, not the question given to the model.

In exact arithmetic, those routes would produce the same result. In finite precision, they can produce a small difference.

A **logit** is the model’s numerical score for a possible next token before those scores become probabilities or a token is selected. Most small logit changes do not alter the selected token because the highest-scoring candidate remains highest.

A near tie between two candidate tokens increases the risk. Under temperature-zero, or greedy, decoding, the server selects the highest-logit token. A small batch-induced change can reverse the order of a near tie and select a different token.

With nonzero-temperature sampling, changed logits alter token probabilities. Sampling then adds a separate source of variation unless the random state and sampling implementation are controlled.

Once one token changes, autoregressive generation follows a new path. The newly selected token changes the context and key-value cache, which stores previous attention information for faster decoding, so every later token distribution is computed from that changed state; even a small numerical difference at one position can produce a visibly different completion or reasoning trace.

Batch invariance matters wherever engineers need to reproduce a result independently of changing traffic.

A benchmark run can change when the evaluator changes batch size, GPU count, GPU type, precision, or serving framework. The [NeurIPS 2025 study on numerical nondeterminism](https://proceedings.neurips.cc/paper_files/paper/2025/hash/f80094a824ba5912d4a2de169c404a40-Abstract-Conference.html) reported up to 9 percentage points of accuracy variation and a 9,000-token difference in response length for DeepSeek-R1-Distill-Qwen-7B across tested runtime configurations, including batch and hardware changes.

Those findings do not establish that every model or hosted API behaves this way. They do show that evaluation configuration can affect results materially, rather than only changing low-order bits.

For a meaningful comparison, record the batch size and scheduling mode along with the model, tokenizer, precision, GPU configuration, and software versions. A single greedy run with a fixed seed is not enough documentation.

Long reasoning generations face greater exposure because they contain more token decisions. Each additional decision provides another opportunity for a small numerical difference to branch the sequence.

A request that passes when run alone but fails under production load is difficult to replay if batch composition is part of the hidden input. Batch invariance helps these tests depend less on unrelated traffic.

It supports:

Compare token IDs before comparing rendered text. Text formatting can hide or introduce differences unrelated to model numerics.

Reinforcement-learning rollouts depend on the policy that generates tokens. If rollout inference uses a different numerical path from the scoring or training path, the generated data can differ from the behavior expected by the training procedure.

This concern also applies to agentic systems. An early token difference can alter a tool call, selected action, or reasoning branch. The [tensor-parallel invariance preprint](https://arxiv.org/abs/2511.17826) separates batch-related variation from variation caused by changing tensor-parallel sizes, which is a useful distinction when training and rollout infrastructure use different distributed configurations.

Suppose a prompt produces one answer when run alone and another when placed beside requests with longer sequences. The difference alone does not indicate prompt contamination or data leakage. Dynamic batching may have changed the execution shape and, therefore, the floating-point reduction order.

Batch invariance guarantees behavioral isolation: unrelated traffic should not change a request merely by changing how the server groups work. Investigate kernels, scheduling, precision, and distributed execution before treating the difference as evidence that another request’s content entered the target request.

A batch-invariant implementation keeps each request on a stable numerical path even when the enclosing batch changes. The direct method is to fix per-request reduction orders and avoid shape-dependent kernel choices that change those reductions.

That design can disable optimizations such as adaptive Split-K execution or custom collective paths. The supported scope must be explicit because a kernel can be invariant across batch composition while remaining sensitive to hardware, parallelism, or software changes.

RMSNorm, a normalization method that scales a hidden-state vector using the root mean square of its values, reduces many values across a hidden dimension.

A small-batch optimization might split one request’s reduction across multiple GPU cores, while a larger batch uses a different partition. The result can differ slightly because the additions occur in a different order.

A batch-invariant RMSNorm kernel preserves the reduction order for each logical row regardless of the surrounding batch size. The goal is stable arithmetic, not merely a stable scheduler decision.

Matrix multiplication computes many dot products, and each dot product is a reduction. Different batch shapes can select different tile layouts, Tensor Core instructions, or Split-K strategies.

A batch-invariant matrix-multiplication path uses the same reduction scheme for each logical output element across supported shapes. This often means giving up some shape-specific optimization. The performance cost depends on the model, GPU, precision, and workload, so no single percentage applies universally.

Attention combines a query with keys and values across sequence positions. Prefill, decoding, chunked processing, prefix caching, and split-key-value strategies can use different partitions for this work.

For each token, a stable attention implementation keeps the reduction order independent of neighboring requests and chunking arrangements. Key-value cache handling must also preserve the same logical partitioning when requests enter or leave a continuous batch.

Tensor parallelism splits model computation across GPUs. All-reduce order, GPU topology, NCCL behavior, custom collectives, and tensor-parallel size can each introduce numerical variation.

A batch-invariant mode therefore does not automatically provide tensor-parallel invariance. The [vLLM implementation notes](https://docs.vllm.ai/en/v0.17.1/features/batch_invariance) describe deterministic kernels and the disabling of some optimizations that can introduce variation, including certain custom all-reduce paths. That behavior remains limited to the documented hardware, models, and framework version.

Run-to-run determinism and batch invariance answer different questions.

For one fixed shape, a kernel can repeat bit-for-bit yet select a different implementation when the shape changes. Workspace conditions, stream usage, toolkit version, and library heuristics can all affect that choice. NVIDIA’s [cuBLAS reproducibility documentation](https://docs.nvidia.com/cuda/cublas/index.html?highlight=CUBLAS_EMULATION_STRATEGY) limits bitwise guarantees by GPU architecture, toolkit version, and execution conditions.

Batch invariance removes batch and scheduling variation as causes of drift. It does not remove every reproducibility variable. A reproducibility claim should name its scope rather than imply universal bitwise identity.

Temperature zero generally selects the highest-logit token. It does not guarantee that every serving configuration computes the same logits.

If two candidate logits are close, a small change in floating-point arithmetic can change greedy selection. A seed helps with sampling randomness, but it cannot repair a difference that occurred before token selection.

Adaptive kernels respond to workload shape to maximize occupancy and throughput, whereas batch-invariant kernels restrict those choices to preserve a stable arithmetic path, trading peak adaptive performance for reproducible numerics.

Three broad strategies are available:

The NeurIPS study found much lower cross-configuration divergence in FP32 than in BF16 or FP16 in its tested settings. That result supports higher precision as a mitigation, not as a universal replacement for batch-invariant kernels.

Emerging preprints explore selective alternatives. [LLM-42](https://arxiv.org/abs/2601.17768) uses verification and rollback under a fixed-shape schedule. [MarginGate](https://arxiv.org/abs/2605.30218) proposes verifying low-margin token decisions, where a small numerical change is more likely to flip the selected token. Both should be treated as preliminary research rather than established solutions.

Test batch invariance against a declared environment and a declared contract. The contract should say whether the requirement is identical logits, identical token IDs, or equivalent final text. Exact logits are the strongest and most demanding target.

For the same request, run it:

Use greedy decoding first, then repeat with seeded nonzero-temperature sampling. Compare token IDs before logits or rendered text where instrumentation allows. Record the first differing token and the top-two logit margin at that position.

Also record:

If the output changes only when batch composition or request order changes on the same stack, investigate batch-dependent kernels and scheduling first.

If the output changes across GPU types, CUDA versions, or tensor-parallel sizes, batch invariance may not be the cause. Change one variable at a time to keep batch effects separate from changes in model files, tokenizer behavior, hardware, or sampling.

A first differing token with a small top-two margin is consistent with numerical sensitivity. It is not proof of that cause, but it provides a useful diagnostic direction. Do not infer how common the behavior is across hosted APIs from local tests because providers may use undisclosed runtimes, batching policies, and model revisions.

The [vLLM v0.17.1 documentation](https://docs.vllm.ai/en/v0.17.1/usage/reproducibility.html) documents batch invariance as a beta feature. At that version, the documented mode is enabled with:

```
export VLLM_BATCH_INVARIANT=1
vllm serve meta-llama/Llama-3.1-8B-Instruct
```

The documented support is limited to specified NVIDIA hardware, models, and same-version, same-hardware conditions. Check the exact framework version before relying on the setting in production.

No. Temperature zero normally selects the highest-logit token, but batching, hardware, kernel selection, or precision can change the logits. Near-tied candidates can therefore produce different greedy tokens.

No. A seed controls pseudorandom sampling, while batch invariance controls sensitivity to surrounding requests and execution shape. Reproducible nonzero-temperature generation requires both a stable random-state policy and controlled numerical execution.

Yes, in the narrow numerical sense. Another request can change batch shape and kernel scheduling without contributing its prompt content to your computation. A changed result is not, by itself, evidence of information sharing.

No. FP32 reduces rounding sensitivity and produced stronger stability in the cited experiments, but it does not guarantee identical execution paths across all batch shapes, hardware, or software configurations. It also increases memory and compute costs.

No. It covers batch-related variation within a declared execution scope. Cross-hardware, cross-version, and cross-parallelism reproducibility requires separate guarantees and testing.

No universal cost exists. The impact depends on the model, workload, GPU, precision, and kernel implementation. Fixed numerical paths can reduce peak performance, while optimized deterministic paths and selective verification can reduce that penalty.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com