{"slug": "how-does-mixture-of-experts-routing-affect-llm-repeatability", "title": "How Does Mixture-of-Experts Routing Affect LLM Repeatability?", "summary": "A new analysis from the Journal of Machine Learning Research explains that Mixture-of-Experts (MoE) routing can cause large language models to produce different outputs across runs due to discrete expert selection amplifying small numerical differences, even at temperature 0. The report details how batch shape, GPU kernels, hardware, precision, and software versions can alter router inputs, leading to different expert paths and autoregressive divergence, and outlines conditions for deterministic repeatability.", "body_md": "### What Is Batch Invariance in LLM Inference?\n\nBatch invariance means a request produces the same inference result when the server runs it alone, alongside other requests, at a different…\n\n[Mixture-of-Experts (MoE) models](https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf) split a large feed-forward network into smaller experts. A router selects only a few experts per token, improving efficiency but adding discrete choices that can amplify small numerical differences.\n\nMoE routing is not inherently nondeterministic. Repeatability requires identical router inputs, tie-breaking, capacity decisions, and numerical operations to produce the same routes and outputs. It can break when batching, kernels, hardware, precision, software versions, or expert-capacity rules change. Temperature 0 removes sampling randomness but does not guarantee identical logits or expert selections.\n\nRepeatability has several levels:\n\nThese levels are related but not equivalent. Two runs can select the same experts while producing slightly different logits because of floating-point operations. Conversely, two runs can produce the same text even if their internal routes differ.\n\nTemperature 0 selects the highest-probability token from the logits generated during that particular run. It does not force two runs to calculate identical logits. If a changed batch shape or GPU kernel produces a slightly different hidden state, the router receives different inputs. A small change near the routing boundary can then select a different expert.\n\nThe central mechanism is:\n\nMoE routing converts a small numerical change into a discrete expert-path change. That changed path can alter later logits and, after one different token is generated, cause autoregressive divergence.\n\nThis mechanism is separate from general LLM variation. Dense models can also produce different results when matrix-multiplication tiling, reduction order, attention kernels, normalization behavior, hardware, quantization, or software versions change. MoE adds routing sensitivity and, in some designs, direct competition among tokens for expert capacity.\n\nA deterministic MoE implementation remains repeatable when it receives identical inputs and uses stable score ordering, deterministic tie handling, fixed capacity rules, and deterministic downstream computation.\n\nA dense transformer feed-forward layer applies the same learned parameters to every token. An MoE layer contains multiple feed-forward networks, or experts, and activates only a small subset for each token.\n\nFor token `t`\n\n, a simplified token-choice MoE layer works as follows:\n\n```\ns_t = h_t W_r\nE_t = \\operatorname{TopK}(s_t, k)\ny_t = \\sum_{e \\in E_t} g_{t,e} f_e(h_t)\n```\n\nHere:\n\n`h_t`\n\n.`W_r`\n\n.`s_t`\n\n.`k`\n\nselected experts form the set `E_t`\n\n.`e`\n\n, `g_{t,e}`\n\ngives its gate weight.`e`\n\nperforms the computation `f_e`\n\n.`y_t`\n\n.The execution sequence is:\n\n`k`\n\noperation selects the highest-scoring experts.The [Switch Transformer paper](https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf) describes this sparse computation as selecting different parameters for different inputs. The repeatability consequence is that top-`k`\n\nselection introduces a discrete decision into an otherwise mostly continuous computation.\n\nTop-`k`\n\nrouting is not random by itself. If router scores, score ordering, tie-breaking behavior, capacity policy, and implementation are identical, the selected expert set is identical.\n\n`k`\n\nBoundary MattersSuppose a token uses four experts. If the fourth-highest score remains above the fifth-highest score, small changes to the score values do not change the route. If those two scores cross, the fourth selected expert is replaced by the fifth.\n\nThe **router margin** measures this distance:\n\n```\nm_t = s_{t,(k)} - s_{t,(k+1)}\n```\n\nA small margin means that the token is close to changing its selected expert set. A large margin indicates a more stable selection under small score perturbations.\n\nA route change does more than slightly modify a weighted sum. Different experts contain different weights and calculate different transformations. Replacing one expert can change the token's hidden state, which changes later-layer activations and next-token logits.\n\nThe decision rule is simple: a small numerical difference matters for routing only when it changes score ordering, tie handling, or capacity outcomes. Most floating-point differences do not cross a routing boundary.\n\nBatching affects MoE inference in two distinct ways. First, the batch can change numerical execution. Second, some MoE implementations make tokens compete for limited expert capacity.\n\nServing engines combine requests into batches to use hardware more efficiently. Changes to batch size, padding pattern, request order, or sequence lengths can alter:\n\nAn unchanged prompt can produce slightly different hidden states when these execution details change. The [Thinking Machines Lab analysis](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) documents the broader batch-invariance problem in operations such as RMSNorm, matrix multiplication, and attention.\n\nThe same request can therefore behave differently when processed:\n\nA changed hidden state produces different router scores. If the scores around the top-`k`\n\ncutoff are close, the selected experts can change. If the scores are well separated, the route remains stable despite the numerical difference.\n\nThis effect is not unique to MoE. Dense models can also produce different numerical results when batch-dependent kernels change. MoE adds a discrete boundary that can magnify the effect.\n\nSome MoE designs assign each expert a maximum number of tokens per batch. The [Switch Transformer research](https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf) defines expert capacity using the number of tokens, the number of experts, and a capacity factor.\n\nIf too many tokens select one expert, later tokens can overflow. Depending on the implementation, an overflowed token might:\n\nThis creates direct batch coupling. A token's realized computation depends on both its own router scores and the other tokens competing for the same expert.\n\nFor example, a prompt processed alone might reach expert 12 without exceeding capacity. The same prompt placed in a batch containing many tokens routed to expert 12 might overflow and follow a different path. Request order can also matter if the dispatcher resolves capacity in order.\n\n[Switch Transformer reported](https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf) dropped-token rates typically below 1% in its specific experiments when load balancing was sufficiently strong. That result does not describe every deployed MoE system. Other implementations use dropless dispatch, larger buffers, expert replication, static allocation, rerouting, or different fallback policies.\n\nMany decoder-only MoE systems use token-choice routing: each token selects its top experts. Expert-choice routing reverses that relationship. Each expert selects its highest-scoring tokens from the available token pool.\n\nThe [Expert Choice Routing paper](https://papers.nips.cc/paper_files/paper/2022/file/2f00ecd787b432c1d36f3de9800728eb-Paper-Conference.pdf) shows that expert selection depends on the available token population: changing that population can change an expert's selected tokens even when a particular token's own hidden state does not.\n\nExpert-choice routing provides a useful contrast, but its original formulation does not transfer unchanged to ordinary autoregressive decoding. It uses a broader token-selection context, and its serving behavior at small batch sizes presents separate challenges. It should not be treated as evidence that most production decoder-only MoE models use this routing scheme.\n\nThe causal chain from serving conditions to output divergence is:\n\n`k`\n\nboundary, resolve a tie differently, or produce a different capacity outcome.The model does not need random sampling for this chain to occur. Greedy decoding still depends on the exact logits produced by the forward pass.\n\nThis is a causal mechanism, not a claim that every route change produces a different completion. A route change might leave the top token unchanged. A logit difference might also remain too small to alter the selected token. Divergence becomes more likely when the next-token logits are close, the route change occurs early, or later computations amplify the difference.\n\nSeveral results establish that MoE repeatability is an operational concern, while leaving some routing-specific questions unresolved.\n\n[Thinking Machines Lab reported](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) 1,000 temperature-0 completions from `Qwen/Qwen3-235B-A22B-Instruct-2507`\n\n. The experiment generated 1,000 tokens per completion and observed:\n\nThe [Qwen model card](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8) identifies this model as an MoE model with a large total parameter count and a smaller activated parameter count, 128 experts, and 8 activated experts per token.\n\nThe result demonstrates that a large MoE model can produce different temperature-0 completions under one serving configuration and become repeatable under batch-invariant execution. It does not prove that changed expert routes caused every divergence. The experiment did not publish expert-route traces or isolate router changes from numerical differences in RMSNorm, matrix multiplication, attention, and other operations.\n\n[NVIDIA's router replay documentation](https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/router_replay.html) treats floating-point routing variation as a reproducibility problem. Replaying a fixed route table locks the selected MoE path across runs.\n\nRoute replay is useful for two purposes:\n\nIt is not a universal determinism switch. Attention, normalization, reductions, communication, and decoding can still vary.\n\n[Switch Transformer](https://www.jmlr.org/papers/volume23/21-0998/21-0998.pdf) provides direct evidence that finite expert capacity changes how tokens are processed when experts receive too many assignments. Its reported sub-1% dropped-token rate applies to its specific architecture and configuration, not to all modern MoE inference systems.\n\nThe [Thinking Machines Lab report](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) measured a Qwen-3-8B serving workload on one GPU with 1,000 sequences of roughly comparable output lengths. It reported:\n\n| Configuration | Time |\n|---|---|\n| Default vLLM | 26 seconds |\n| Unoptimized deterministic vLLM | 55 seconds |\n| Deterministic vLLM with an improved attention kernel | 42 seconds |\n\nThese measurements show a trade-off rather than a universal performance penalty. Deterministic or batch-invariant execution can reduce throughput and scheduling flexibility, although optimized deterministic kernels can recover part of the cost.\n\nPublic evidence does not yet provide a broad benchmark covering:\n\nTherefore, output variation in an MoE model should not automatically be attributed to expert routing.\n\nMoE routing explains one path to non-repeatability, not every path.\n\n**MoE is not synonymous with nondeterminism.** Fixed router inputs and deterministic execution can produce identical routes.\n\n**Batch sensitivity is not exclusive to MoE.** Dense models also experience batch-dependent numerical behavior. MoE adds route boundaries and, in some architectures, capacity competition.\n\n**Top- k routing is not inherently random.** Variation requires changed scores, different tie handling, changed capacity outcomes, or another implementation difference.\n\n**Not every MoE system uses finite expert capacity.** Dropless systems and systems with different allocation policies do not behave like the Switch Transformer design.\n\n**Identical text does not prove identical routes.** Two internal computations can produce the same argmax token at every step. Route identity requires router telemetry or replay.\n\n**A fixed seed is insufficient.** A seed controls sampling state where supported. It does not fix model weights, kernels, batch composition, hardware behavior, quantization, software versions, or router inputs.\n\nRun repeatability tests in layers so that each result answers a specific question.\n\nRun many identical requests with temperature 0 and record:\n\nRepeat the test with the request processed alone, at different batch sizes, with different request orderings, and under realistic co-tenant load.\n\nCompare normal serving with:\n\nThe [vLLM batch-invariance documentation](https://docs.vllm.ai/en/stable/features/batch_invariance) describes a mode intended to make outputs independent of batch size and request order for supported configurations. Its [reproducibility guidance](https://docs.vllm.ai/en/v0.17.1/usage/reproducibility.html) also limits repeatability claims to the same hardware and vLLM version.\n\nWhen the serving stack exposes the required telemetry, record:\n\nCalculate the router margin between the `k`\n\n-th and `(k+1)`\n\n-th scores. Tokens with small margins deserve focused investigation because small numerical changes are more likely to alter their expert set.\n\nUse the first divergence in each trace:\n\nCompare the first route difference, first logit difference, and first output-token difference separately. Identical routes argue against route changes, but they do not prove bitwise-identical execution elsewhere.\n\nThe appropriate control depends on the required repeatability target. Token-exact regression tests need stricter controls than production workloads that tolerate semantic variation.\n\nRecord temperature, top-p, top-k, repetition penalties, maximum tokens, stopping rules, and seed settings. Keep prompts, retrieved documents, tool results, timestamps, and external context identical.\n\nTemperature 0 is appropriate for testing greedy decoding, but it does not replace execution controls.\n\nRecord the exact:\n\nScores may change after a runtime update or quantization change, even with the same prompt.\n\nFor token-exact tests, constrain dynamic batching and use deterministic or batch-invariant kernels where supported. Test both fixed laboratory batches and varied production-like batches. This separates reproducibility under controlled conditions from robustness to serving conditions.\n\nExpect a performance trade-off. Deterministic execution can reduce throughput, increase latency, or limit scheduling flexibility.\n\nLog expert IDs, router scores, gate weights, and capacity decisions when possible. Use route replay as a debugging or ablation tool, not as a complete production solution. A fixed route does not force attention, normalization, matrix reductions, distributed communication, or final decoding to be bitwise identical.\n\nChoose one of these targets before testing:\n\nScientific experiments, regression tests, and reinforcement-learning rollouts often need token-exact behavior. Many production systems instead need documented variation limits and stable task-level quality.\n\nVisible text alone cannot identify the cause of variation. A route-aware comparison should align repeated runs by token position and MoE layer, then compare:\n\nThe first changed expert set marks route drift. The first changed router scores without a route change indicate numerical drift that did not cross a selection boundary. A changed route alongside stable pre-router values points to dispatch, tie handling, or capacity behavior.\n\nRoute traces provide stronger evidence than output comparison, but identical route traces still do not prove that every operation was bitwise identical. They show only that the selected MoE path matched.\n\nNo. At temperature 0, each run selects the highest-logit token from its logits. Changes in kernels, batch composition, hardware, software, or expert routes can alter those logits and the resulting greedy output.\n\nUsually not. Top-`k`\n\nrouting is deterministic for fixed scores, tie-breaking rules, capacity policies, and implementation behavior. Repeatability problems arise when those inputs or conditions change.\n\nYes, depending on the serving implementation. Other requests can change how numerical kernels behave, and capacity-limited designs can make tokens compete directly for expert slots.\n\nNo. Route replay fixes the selected MoE path, but other operations can still produce different values. Use it alongside pinned software and hardware, deterministic kernels, controlled batching, and fixed decoding settings.\n\nGive Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.\n\nTake a look at vroni.com", "url": "https://wpnews.pro/news/how-does-mixture-of-experts-routing-affect-llm-repeatability", "canonical_source": "https://www.vincentschmalbach.com/mixture-of-experts-routing-llm-repeatability/", "published_at": "2026-08-11 09:13:46+00:00", "updated_at": "2026-08-11 09:19:47.314445+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research"], "entities": ["Journal of Machine Learning Research", "Mixture-of-Experts", "Switch Transformer"], "alternates": {"html": "https://wpnews.pro/news/how-does-mixture-of-experts-routing-affect-llm-repeatability", "markdown": "https://wpnews.pro/news/how-does-mixture-of-experts-routing-affect-llm-repeatability.md", "text": "https://wpnews.pro/news/how-does-mixture-of-experts-routing-affect-llm-repeatability.txt", "jsonld": "https://wpnews.pro/news/how-does-mixture-of-experts-routing-affect-llm-repeatability.jsonld"}}