What Is Batch Invariance in LLM Inference?
Batch invariance means a request produces the same inference result when the server runs it alone, alongside other requests, at a different…
Mixture-of-Experts (MoE) models split a large feed-forward network into smaller experts. A router selects only a few experts per token, improving efficiency but adding discrete choices that can amplify small numerical differences.
MoE routing is not inherently nondeterministic. Repeatability requires identical router inputs, tie-breaking, capacity decisions, and numerical operations to produce the same routes and outputs. It can break when batching, kernels, hardware, precision, software versions, or expert-capacity rules change. Temperature 0 removes sampling randomness but does not guarantee identical logits or expert selections.
Repeatability has several levels:
These levels are related but not equivalent. Two runs can select the same experts while producing slightly different logits because of floating-point operations. Conversely, two runs can produce the same text even if their internal routes differ.
Temperature 0 selects the highest-probability token from the logits generated during that particular run. It does not force two runs to calculate identical logits. If a changed batch shape or GPU kernel produces a slightly different hidden state, the router receives different inputs. A small change near the routing boundary can then select a different expert.
The central mechanism is:
MoE routing converts a small numerical change into a discrete expert-path change. That changed path can alter later logits and, after one different token is generated, cause autoregressive divergence.
This mechanism is separate from general LLM variation. Dense models can also produce different results when matrix-multiplication tiling, reduction order, attention kernels, normalization behavior, hardware, quantization, or software versions change. MoE adds routing sensitivity and, in some designs, direct competition among tokens for expert capacity.
A deterministic MoE implementation remains repeatable when it receives identical inputs and uses stable score ordering, deterministic tie handling, fixed capacity rules, and deterministic downstream computation.
A dense transformer feed-forward layer applies the same learned parameters to every token. An MoE layer contains multiple feed-forward networks, or experts, and activates only a small subset for each token.
For token t
, a simplified token-choice MoE layer works as follows:
s_t = h_t W_r
E_t = \operatorname{TopK}(s_t, k)
y_t = \sum_{e \in E_t} g_{t,e} f_e(h_t)
Here:
h_t
.W_r
.s_t
.k
selected experts form the set E_t
.e
, g_{t,e}
gives its gate weight.e
performs the computation f_e
.y_t
.The execution sequence is:
k
operation selects the highest-scoring experts.The Switch Transformer paper describes this sparse computation as selecting different parameters for different inputs. The repeatability consequence is that top-k
selection introduces a discrete decision into an otherwise mostly continuous computation.
Top-k
routing is not random by itself. If router scores, score ordering, tie-breaking behavior, capacity policy, and implementation are identical, the selected expert set is identical.
k
Boundary MattersSuppose a token uses four experts. If the fourth-highest score remains above the fifth-highest score, small changes to the score values do not change the route. If those two scores cross, the fourth selected expert is replaced by the fifth.
The router margin measures this distance:
m_t = s_{t,(k)} - s_{t,(k+1)}
A small margin means that the token is close to changing its selected expert set. A large margin indicates a more stable selection under small score perturbations.
A route change does more than slightly modify a weighted sum. Different experts contain different weights and calculate different transformations. Replacing one expert can change the token's hidden state, which changes later-layer activations and next-token logits.
The decision rule is simple: a small numerical difference matters for routing only when it changes score ordering, tie handling, or capacity outcomes. Most floating-point differences do not cross a routing boundary.
Batching affects MoE inference in two distinct ways. First, the batch can change numerical execution. Second, some MoE implementations make tokens compete for limited expert capacity.
Serving engines combine requests into batches to use hardware more efficiently. Changes to batch size, padding pattern, request order, or sequence lengths can alter:
An unchanged prompt can produce slightly different hidden states when these execution details change. The Thinking Machines Lab analysis documents the broader batch-invariance problem in operations such as RMSNorm, matrix multiplication, and attention.
The same request can therefore behave differently when processed:
A changed hidden state produces different router scores. If the scores around the top-k
cutoff are close, the selected experts can change. If the scores are well separated, the route remains stable despite the numerical difference.
This effect is not unique to MoE. Dense models can also produce different numerical results when batch-dependent kernels change. MoE adds a discrete boundary that can magnify the effect.
Some MoE designs assign each expert a maximum number of tokens per batch. The Switch Transformer research defines expert capacity using the number of tokens, the number of experts, and a capacity factor.
If too many tokens select one expert, later tokens can overflow. Depending on the implementation, an overflowed token might:
This creates direct batch coupling. A token's realized computation depends on both its own router scores and the other tokens competing for the same expert.
For example, a prompt processed alone might reach expert 12 without exceeding capacity. The same prompt placed in a batch containing many tokens routed to expert 12 might overflow and follow a different path. Request order can also matter if the dispatcher resolves capacity in order.
Switch Transformer reported dropped-token rates typically below 1% in its specific experiments when load balancing was sufficiently strong. That result does not describe every deployed MoE system. Other implementations use dropless dispatch, larger buffers, expert replication, static allocation, rerouting, or different fallback policies.
Many decoder-only MoE systems use token-choice routing: each token selects its top experts. Expert-choice routing reverses that relationship. Each expert selects its highest-scoring tokens from the available token pool.
The Expert Choice Routing paper shows that expert selection depends on the available token population: changing that population can change an expert's selected tokens even when a particular token's own hidden state does not.
Expert-choice routing provides a useful contrast, but its original formulation does not transfer unchanged to ordinary autoregressive decoding. It uses a broader token-selection context, and its serving behavior at small batch sizes presents separate challenges. It should not be treated as evidence that most production decoder-only MoE models use this routing scheme.
The causal chain from serving conditions to output divergence is:
k
boundary, resolve a tie differently, or produce a different capacity outcome.The model does not need random sampling for this chain to occur. Greedy decoding still depends on the exact logits produced by the forward pass.
This is a causal mechanism, not a claim that every route change produces a different completion. A route change might leave the top token unchanged. A logit difference might also remain too small to alter the selected token. Divergence becomes more likely when the next-token logits are close, the route change occurs early, or later computations amplify the difference.
Several results establish that MoE repeatability is an operational concern, while leaving some routing-specific questions unresolved.
Thinking Machines Lab reported 1,000 temperature-0 completions from Qwen/Qwen3-235B-A22B-Instruct-2507
. The experiment generated 1,000 tokens per completion and observed:
The Qwen model card identifies this model as an MoE model with a large total parameter count and a smaller activated parameter count, 128 experts, and 8 activated experts per token.
The result demonstrates that a large MoE model can produce different temperature-0 completions under one serving configuration and become repeatable under batch-invariant execution. It does not prove that changed expert routes caused every divergence. The experiment did not publish expert-route traces or isolate router changes from numerical differences in RMSNorm, matrix multiplication, attention, and other operations.
NVIDIA's router replay documentation treats floating-point routing variation as a reproducibility problem. Replaying a fixed route table locks the selected MoE path across runs.
Route replay is useful for two purposes:
It is not a universal determinism switch. Attention, normalization, reductions, communication, and decoding can still vary.
Switch Transformer provides direct evidence that finite expert capacity changes how tokens are processed when experts receive too many assignments. Its reported sub-1% dropped-token rate applies to its specific architecture and configuration, not to all modern MoE inference systems.
The Thinking Machines Lab report measured a Qwen-3-8B serving workload on one GPU with 1,000 sequences of roughly comparable output lengths. It reported:
| Configuration | Time |
|---|---|
| Default vLLM | 26 seconds |
| Unoptimized deterministic vLLM | 55 seconds |
| Deterministic vLLM with an improved attention kernel | 42 seconds |
These measurements show a trade-off rather than a universal performance penalty. Deterministic or batch-invariant execution can reduce throughput and scheduling flexibility, although optimized deterministic kernels can recover part of the cost.
Public evidence does not yet provide a broad benchmark covering:
Therefore, output variation in an MoE model should not automatically be attributed to expert routing.
MoE routing explains one path to non-repeatability, not every path.
MoE is not synonymous with nondeterminism. Fixed router inputs and deterministic execution can produce identical routes.
Batch sensitivity is not exclusive to MoE. Dense models also experience batch-dependent numerical behavior. MoE adds route boundaries and, in some architectures, capacity competition.
Top- k routing is not inherently random. Variation requires changed scores, different tie handling, changed capacity outcomes, or another implementation difference.
Not every MoE system uses finite expert capacity. Dropless systems and systems with different allocation policies do not behave like the Switch Transformer design.
Identical text does not prove identical routes. Two internal computations can produce the same argmax token at every step. Route identity requires router telemetry or replay.
A fixed seed is insufficient. A seed controls sampling state where supported. It does not fix model weights, kernels, batch composition, hardware behavior, quantization, software versions, or router inputs.
Run repeatability tests in layers so that each result answers a specific question.
Run many identical requests with temperature 0 and record:
Repeat the test with the request processed alone, at different batch sizes, with different request orderings, and under realistic co-tenant load.
Compare normal serving with:
The vLLM batch-invariance documentation describes a mode intended to make outputs independent of batch size and request order for supported configurations. Its reproducibility guidance also limits repeatability claims to the same hardware and vLLM version.
When the serving stack exposes the required telemetry, record:
Calculate the router margin between the k
-th and (k+1)
-th scores. Tokens with small margins deserve focused investigation because small numerical changes are more likely to alter their expert set.
Use the first divergence in each trace:
Compare the first route difference, first logit difference, and first output-token difference separately. Identical routes argue against route changes, but they do not prove bitwise-identical execution elsewhere.
The appropriate control depends on the required repeatability target. Token-exact regression tests need stricter controls than production workloads that tolerate semantic variation.
Record temperature, top-p, top-k, repetition penalties, maximum tokens, stopping rules, and seed settings. Keep prompts, retrieved documents, tool results, timestamps, and external context identical.
Temperature 0 is appropriate for testing greedy decoding, but it does not replace execution controls.
Record the exact:
Scores may change after a runtime update or quantization change, even with the same prompt.
For token-exact tests, constrain dynamic batching and use deterministic or batch-invariant kernels where supported. Test both fixed laboratory batches and varied production-like batches. This separates reproducibility under controlled conditions from robustness to serving conditions.
Expect a performance trade-off. Deterministic execution can reduce throughput, increase latency, or limit scheduling flexibility.
Log expert IDs, router scores, gate weights, and capacity decisions when possible. Use route replay as a debugging or ablation tool, not as a complete production solution. A fixed route does not force attention, normalization, matrix reductions, distributed communication, or final decoding to be bitwise identical.
Choose one of these targets before testing:
Scientific experiments, regression tests, and reinforcement-learning rollouts often need token-exact behavior. Many production systems instead need documented variation limits and stable task-level quality.
Visible text alone cannot identify the cause of variation. A route-aware comparison should align repeated runs by token position and MoE layer, then compare:
The first changed expert set marks route drift. The first changed router scores without a route change indicate numerical drift that did not cross a selection boundary. A changed route alongside stable pre-router values points to dispatch, tie handling, or capacity behavior.
Route traces provide stronger evidence than output comparison, but identical route traces still do not prove that every operation was bitwise identical. They show only that the selected MoE path matched.
No. At temperature 0, each run selects the highest-logit token from its logits. Changes in kernels, batch composition, hardware, software, or expert routes can alter those logits and the resulting greedy output.
Usually not. Top-k
routing is deterministic for fixed scores, tie-breaking rules, capacity policies, and implementation behavior. Repeatability problems arise when those inputs or conditions change.
Yes, depending on the serving implementation. Other requests can change how numerical kernels behave, and capacity-limited designs can make tokens compete directly for expert slots.
No. Route replay fixes the selected MoE path, but other operations can still produce different values. Use it alongside pinned software and hardware, deterministic kernels, controlled batching, and fixed decoding settings.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com