Recently I read aboyt this article:
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
Here is what I learnt:
LLM inference engines provide a platform for running large language models on user hardware. However, the choice of inference engine can significantly affect model behaviour, introducing variations in output quality, reasoning length, and overall performance.
Beyond the model itself, the inference stack introduces additional factors that can influence results:
Prefix caching mechanisms are enabled by default in inference engines such as vLLM, Ollama, and SGLang. Although they are designed to reuse previously computed KV cache for improved efficiency, they can alter execution paths and the order of floating-point operations, leading to minor numerical variations.
Kernel-level tie-breaking during greedy decoding. When multiple tokens have identical logits (particularly under FP16 precision), PyTorch deterministically selects the lowest token ID, whereas LMDeploy's parallel Top-K kernel may select different tokens due to thread scheduling, resulting in divergent generation.
Accumulation precision during matrix multiplication. Inference engines differ in their choice of accumulation precision. For example, llama.cpp and Ollama may accumulate intermediate matrix multiplication results in FP32, whereas other engines perform accumulation in FP16. These differences introduce small rounding variations that can propagate through subsequent transformer layers and ultimately affect the generated output.
These factors are fundamentally manifestations of the same underlying issue: the inference stack is not merely an execution layer but part of the model's effective runtime environment. Small implementation differences can lead to measurable variations in generated outputs, especially for large reasoning models.
Therefore, simply fixing the random seed is insufficient to guarantee reproducibility. Deterministic behaviour requires controlling and documenting the complete inference stack, including:
The inference environment should be treated as part of the model specification, and the complete system footprint should be exposed alongside evaluation results to ensure reproducibility and fair comparison between deployments.
Credit goes to the original authors. Their paper provides more detailed explanations, methodology, and comprehensive results that inspired this writeup.