{"slug": "does-setting-a-seed-make-llm-output-reproducible", "title": "Does Setting a Seed Make LLM Output Reproducible?", "summary": "Setting a seed does not guarantee identical large language model (LLM) output, according to an analysis of LLM inference. A seed initializes the pseudorandom number generator used in token sampling but does not lock the model, prompt construction, hardware, numerical calculations, or hosted infrastructure. Microsoft documents Azure OpenAI seeded generation as a best-effort attempt at deterministic sampling, not a guarantee, and an archived OpenAI Cookbook example tied repeatability to matching request parameters and system_fingerprint.", "body_md": "### What Is Batch Invariance in LLM Inference?\n\nBatch invariance means a request produces the same inference result when the server runs it alone, alongside other requests, at a different…\n\nNo. A seed can improve repeatability, but it does not guarantee identical large language model (LLM) output. It initializes the pseudorandom number generator used in token sampling; it does not lock the model, prompt construction, hardware, numerical calculations, or hosted infrastructure.\n\nTo replay output exactly, you usually need the same model revision, tokenizer and chat template, fully rendered prompt, decoding settings, random state, runtime, hardware behavior, batching conditions, and dynamic inputs. Hosted APIs generally offer best-effort repeatability. Self-hosted deployments can provide stronger guarantees only when engineers pin and verify the full inference environment.\n\nAn LLM generates text one token at a time. At each step, the model calculates a probability for every possible next token. The decoder then selects one token from that distribution.\n\nWith sampling enabled, decoding parameters determine how the distribution is used:\n\nThe seed initializes the pseudorandom sequence used to select tokens from the resulting distribution. If the model produces the same distribution at every step, the same seed and decoding configuration produce the same sampling choices.\n\nThat is why engineers should still set seeds for debugging, prompt comparisons, regression tests, and controlled evaluations. A seed removes avoidable sampling variance. It does not control the entire function that produces the output:\n\n```\n\\text{output} = f(\\text{weights}, \\text{prompt}, \\text{tokenizer}, \\text{decoding}, \\text{random state}, \\text{runtime})\n```\n\nA numeric seed has meaning only within a particular implementation. Different providers and inference engines can use different pseudorandom generators, tokenizers, sampling algorithms, and seed semantics.\n\nA seed identifies neither the model weights nor the probability distribution from which the decoder samples. Reusing seed `42`\n\nwith another model does not reproduce the first model’s choices. Changing the model deployment, prompt template, tokenizer, or external input also changes the token probabilities, so the same random sequence no longer produces the same response.\n\nMicrosoft documents Azure OpenAI seeded generation as a [best-effort attempt at deterministic sampling](https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/reproducible-output), not a guarantee. An [archived OpenAI Cookbook example](https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter) similarly tied repeatability to matching request parameters and `system_fingerprint`\n\n, and applied its guidance to specific 2023 model versions rather than to all current models.\n\nTreat a seed as one field in a replay record, not as a portable identifier for a response.\n\nTemperature zero normally selects the highest-probability token instead of deliberately sampling among alternatives. This is called **greedy decoding**. Greedy decoding removes one source of randomness, but it does not guarantee that the model computes exactly the same probabilities on every run.\n\nThe model calculates logits, numerical scores for each possible next token. Floating-point arithmetic rounds those scores. [Parallel GPU operations](https://proceedings.neurips.cc/paper_files/paper/2025/file/f80094a824ba5912d4a2de169c404a40-Paper-Conference.pdf) can also combine values in different orders. Because floating-point addition is not perfectly associative, changing the operation order, numerical precision, kernel, batch, or hardware can slightly change a logit.\n\nIf two tokens have nearly equal scores, a small numerical difference can reverse their ranking. Once one token changes, the model receives a different sequence on the next step. The remaining completion can then diverge substantially.\n\nA seed has little direct effect during greedy decoding because the decoder selects an argmax, or highest-scoring token. The seed cannot correct a changed argmax.\n\nSampling randomness and numerical nondeterminism are separate problems.\n\nA fixed seed repeats the same random draw only when the decoder uses the same probability distribution. If rounding or a different execution path changes that distribution, the same random stream can select a different token.\n\nTemperature zero removes the random draw but still depends on stable token scores. Therefore, deterministic random-number generation and deterministic model computation require separate engineering controls.\n\nA serving engine combines multiple requests into a batch to improve hardware utilization. Different batch contents, request order, padding, or scheduler decisions can select different GPU kernels or reduction orders. The visible prompt stays the same, but the arithmetic path changes.\n\nThe [Thinking Machines Lab report on LLM nondeterminism](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) describes this mechanism and reports an experiment in which a Qwen model generated 80 unique completions across 1,000 temperature-zero attempts under default settings. With batch-invariant kernels enabled, all 1,000 completions matched. This is engineering evidence from one implementation, not a universal benchmark, but it demonstrates why serving behavior matters.\n\nExact reproduction requires more than matching the visible prompt and seed. The complete execution contract includes the following variables.\n\nRecord and pin:\n\nA provider can update a backend without changing the model name. A local deployment can also change output after an engine upgrade, a new quantization artifact, or a modified chat template.\n\nThe effective prompt includes more than the user message. Preserve:\n\nTwo requests that look identical in an application log can produce different prompts if retrieval results, timestamps, tool responses, or formatting steps differ.\n\nRecord every generation control, including:\n\nChanging a single control changes the probability distribution or the point at which generation stops.\n\nThe serving layer can change computation through:\n\nHosted API callers generally cannot freeze these variables. A self-hosted deployment can control more of them, but only if its serving mode exposes the necessary settings.\n\nFor local inference, record:\n\nDifferent GPU counts or precision formats can produce different arithmetic paths even with identical weights and prompts.\n\nTwo 2025 studies show why a fixed seed and temperature zero are insufficient as universal guarantees.\n\nThe ACL paper [ Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments](https://aclanthology.org/2025.eval4nlp-1.12.pdf) tested five hosted LLM systems across eight tasks. The researchers used identical inputs,\n\n`temperature=0`\n\n, `top_p=1`\n\n, and fixed seeds, with ten runs per condition. The experiments took place in February 2025.The study found accuracy variation of up to 15 percentage points across repeated runs under settings intended to be deterministic. It also reported a particularly large maximum-to-minimum gap for Mixtral-8x7B on one college-math task, where accuracy ranged from 75% to 3% across the tested runs.\n\nRaw text agreement was often lower than parsed-answer agreement. Different explanations or formatting could still contain the same selected answer, while exact-string comparisons treated them as different. This distinction matters for downstream parsers, evaluators, and snapshot tests.\n\nThe study covered only five systems, eight tasks, and ten runs per condition, so its measured variation does not establish a failure rate for every provider or workload. It does establish that fixed seeds and zero temperature do not guarantee stable hosted output.\n\nThe NeurIPS 2025 study, [ Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](https://proceedings.neurips.cc/paper_files/paper/2025/file/f80094a824ba5912d4a2de169c404a40-Paper-Conference.pdf), varied GPU type, GPU count, batch size, and numerical precision.\n\nFor DeepSeek-R1-Distill-Qwen-7B under BF16 precision, the researchers reported up to 9 percentage points of accuracy variation across tested runtime configurations. On MATH500, more than 90% of examples diverged under BF16 across those configurations, compared with 2.2% under FP32. They also observed response-length differences of up to 9,000 tokens in tested cases.\n\nThe results do not mean that FP32 guarantees determinism in every deployment. They show that lower-precision arithmetic and runtime configuration can materially affect greedy decoding. The same seed cannot compensate for a different numerical path.\n\nHosted APIs are convenient but expose only part of the reproducibility contract. A provider might expose a model identifier, seed, request parameters, request ID, or backend fingerprint. The caller generally cannot freeze the provider’s GPU type, batching, scheduler, kernel libraries, numerical precision, routing, or deployment revision.\n\nRecord the provider’s available metadata, especially a system or backend fingerprint. [Microsoft recommends monitoring system_fingerprint](https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/reproducible-output) because backend changes can affect repeatability. A matching fingerprint can provide useful evidence that the backend configuration did not change, but it is not proof that every internal computation was identical.\n\nSelf-hosting provides stronger control, not automatic determinism. vLLM’s [version 0.10.2 reproducibility documentation](https://docs.vllm.ai/en/v0.10.2/usage/reproducibility.html) conditions reproducibility on using the same hardware and vLLM version and warns that ordinary online serving does not guarantee reproducibility. The documentation describes its newer [batch-invariance feature](https://docs.vllm.ai/en/stable/features/batch_invariance) as a beta feature designed to reduce dependence on batch size and request order, and limits it to supported NVIDIA hardware.\n\nChoose the deployment model based on the required level of repeatability:\n\n“Reproducible” describes several different targets. Choose the target before designing a test.\n\n| Target | Meaning | Example test |\n|---|---|---|\n| Bitwise reproducibility | Intermediate computations and output bytes match | Compare recorded tensors and output hashes |\n| String-level repeatability | Returned text matches exactly | Compare the two response strings |\n| Answer-level repeatability | Parsed answers or structured values match | Compare labels, fields, or extracted values |\n| Semantic consistency | Responses preserve the same material meaning | Use task-specific review or a validated metric |\n| Statistical reproducibility | Aggregate results remain within a reported range | Compare means, variance, or confidence intervals |\n\nA system can produce identical text and still produce an incorrect answer. It can also produce different wording while preserving the same parsed JSON value or classification label.\n\nFor JSON extraction, measure schema validity, parser success, field-level agreement, and required-value accuracy. Raw string equality is a poor primary metric if whitespace, field order, or explanatory text does not affect the downstream result.\n\nFor classification, repeat representative inputs and measure label disagreement, accuracy, and calibration. A stable label is not enough if the model is consistently wrong.\n\nFor assistants, evaluate task success, safety behavior, required facts, and important constraints across repeated runs. For benchmarks, report repeated-run distributions rather than one score when exact replay is not assured.\n\nFor audit-sensitive workflows, store the original request, response, model metadata, dynamic inputs, and evaluation result together. That record supports both exact comparison and task-level investigation.\n\nFor a hosted API, log at least:\n\nSnapshot dynamic inputs rather than relying on the original request alone. A changed retrieval result or tool response changes the effective prompt even when the user message remains unchanged.\n\nFor self-hosted inference, pin the model and tokenizer revisions, chat template, quantization artifacts, inference-engine commit, CUDA and driver versions, kernel libraries, GPU topology, precision, and scheduler configuration. Control or record batch size, concurrency, request order, speculative decoding, caching, tensor parallelism, and prefill behavior.\n\nThen run a representative regression suite repeatedly under production-like concurrency. Compare output hashes for string-level changes and task-level metrics for meaningful behavior changes. A seed belongs in that audit trail, but it is not evidence that the full system is replayable.\n\nSetting a seed is useful because it reduces sampling variance and makes debugging and regression testing more controlled. It does not make LLM output inherently reproducible.\n\nFor exact text, freeze and verify the complete model-serving environment. For most production systems, test parsed answers, labels, schema validity, task success, safety behavior, or statistical performance across repeated runs instead.\n\nNo. A seed controls pseudorandom choices during sampling, but model updates, backend behavior, numerical computation, batching, and dynamic inputs remain outside its control.\n\nThey often match on a stable deployment with fixed decoding settings, but exact equality is not guaranteed. Treat repeated matching results as evidence of observed repeatability, and consult provider or serving-stack documentation for any stated guarantees.\n\nNo. Temperature zero removes deliberate sampling randomness through greedy decoding. Numerical rounding, GPU execution paths, batching, scheduling, or backend changes can still alter the highest-scoring token.\n\nNo. Providers can use different tokenizers, model distributions, pseudorandom generators, sampling implementations, and seed semantics. A seed is provider-specific rather than a portable replay key.\n\nGive Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.\n\nTake a look at vroni.com", "url": "https://wpnews.pro/news/does-setting-a-seed-make-llm-output-reproducible", "canonical_source": "https://www.vincentschmalbach.com/does-setting-a-seed-make-llm-output-reproducible/", "published_at": "2026-08-11 09:12:11+00:00", "updated_at": "2026-08-11 09:20:00.782216+00:00", "lang": "en", "topics": ["large-language-models", "ai-research"], "entities": ["Microsoft", "Azure OpenAI", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/does-setting-a-seed-make-llm-output-reproducible", "markdown": "https://wpnews.pro/news/does-setting-a-seed-make-llm-output-reproducible.md", "text": "https://wpnews.pro/news/does-setting-a-seed-make-llm-output-reproducible.txt", "jsonld": "https://wpnews.pro/news/does-setting-a-seed-make-llm-output-reproducible.jsonld"}}