Why does a local LLM give a different score every time, and does temperature 0 fix it?

A developer measured how much a local LLM's CV-scoring output varies by reading the model's raw next-token probabilities, using Qwen2.5-3B-Instruct (4-bit, MLX) on an 8 GB M1 Mac. A single forward pass over a pre-filled "Score: " prompt showed the model assigns 74.2% probability to a 7 and 23.3% to a 6 at temperature 1, and that llama.cpp's default sampler chain (top-k 40, top-p 0.95, min-p 0.05, temp 0.8) narrows the choice to just those two scores. The work also shows what temperature 0 does not fix, since the underlying distribution remains probabilistic.

Give a language model a CV and ask for a score from 1 to 10, and you get a number and a confident reason. Ask again and the number can move. The usual advice is "set temperature to 0", which is correct but leaves two questions open: how much does the score actually move with default settings, and what does temperature 0 not fix? Both can be measured on a laptop in a few minutes, because the score is a single token and the model hands you its probability directly. The setup below uses Qwen2.5-3B-Instruct 4-bit, MLX with mlx-lm 0.32.2 on an 8 GB M1 Mac. The prompts are the ones from a CV-scoring demo in the osFoundry desktop app's Models → Playground: a six-line CV for a junior data analyst role, followed by "Score this CV from 1 to 10 for a junior data analyst role. Start with the score, then one reason." The app runs local models through llama.cpp's llama-server , so the sampler settings come from there. When a request carries no temperature, llama-server samples with its own defaults. The build bundled with the app version 10358 lists them in --help : --temp, --temperature N temperature default: 0.80 --top-k N top-k sampling default: 40, 0 = disabled --top-p N top-p sampling default: 0.95, 1.0 = disabled --min-p N min-p sampling default: 0.05, 0.0 = disabled -s, --seed SEED RNG seed default: -1, use random seed for -1 These apply to every request that doesn't set its own values. When the app starts llama-server, it only adds --temp if a temperature has been configured for the server local inference.rs , so a request without a temperature samples at 0.8 with a random seed, which is also what most llama.cpp front ends do out of the box. A free-form answer can start with "Score: 7", " 7/10 " or "I would rate this CV a 7", which makes the score hard to locate. The fix is to pre-fill the start of the reply, so the very next token the model produces is the digit: python import mlx.core as mx from mlx lm import load, generate from mlx lm.sample utils import make sampler model, tok = load "mlx-community/Qwen2.5-3B-Instruct-4bit" DIGITS = {d: tok.encode str d 0 for d in range 1, 10 } def ids for cv, question, prefill="Score: " : msgs = {"role": "user", "content": cv + "\n\n" + question} text = tok.apply chat template msgs, add generation prompt=True, tokenize=False return tok.encode text + prefill def digit probs ids, T=1.0 : logits = model mx.array ids 0, -1 .astype mx.float32 p = mx.softmax logits / T return {d: float p t for d, t in DIGITS.items } Qwen's tokenizer splits numbers into single digits, so "7" is one token and "10" would start with the token "1". One forward pass over the 173-token prompt gives the whole distribution of the score. For the English CV, at temperature 1 the model's raw probabilities : | Score | Probability | |---|---| | 7 | 74.2% | | 6 | 23.3% | | 8 | 1.3% | | 5 | 1.2% | The model doesn't "think" the CV is a 7. It thinks it is probably a 7, with a one-in-four chance of a 6. Temperature divides the logits before the softmax, which sharpens or flattens this: | Temperature | P 7 | P 6 | P 8 | |---|---|---|---| | 1.0 | 74.2% | 23.3% | 1.3% | | 0.8 | 80.2% | 18.9% | 0.5% | | 0.5 | 91.0% | 9.0% | 0.0% | | 0.2 | 99.7% | 0.3% | 0.0% | llama-server doesn't apply temperature alone. Its default chain runs top-k 40, then top-p 0.95, then min-p 0.05, and temperature last. Applied to these logits, top-p already cuts everything after 7 and 6 together they are over 95% , so the default sampler picks between exactly two scores: python def llamacpp default logits, temp=0.8, top k=40, top p=0.95, min p=0.05 : order = mx.argsort -logits :top k kept = logits order p = mx.softmax kept n = int mx.sum mx.cumsum p < top p .item + 1 idx = i for i in range n if p i .item = min p p 0 .item q = mx.softmax kept mx.array idx / temp return {tok.decode int order i : float q j for j, i in enumerate idx } - {'7': 0.8093, '6': 0.1907} Then I sampled it for real: 200 generations from the same prompt with mlx-lm's sampler set to the same four values make sampler temp=0.8, top p=0.95, min p=0.05, top k=40 . | | 7 | 6 | |---|---|---| | Predicted by the chain | 80.9% | 19.1% | | Observed, 200 runs | 162 81% | 38 19% | So with default settings, about one run in five scores this CV a 6 instead of a 7. If you score two candidates once each and one gets a 6 and the other a 7, there is a real chance the gap is the draw. The same CV and question in Japanese 222 tokens, pre-fill 「点数：」 gave a much less decided model: | Score | Probability at T=1 | Observed, 200 runs at the defaults | |---|---|---| | 8 | 39.8% | 108 | | 9 | 23.8% | 44 | | 7 | 19.4% | 40 | | 10 | starts with "1": 4.1% | 8 | Here the most likely score wins only about half the time, and four different scores came out of 200 runs. How much a score wobbles is not a fixed property of the model. It depends on how close the top candidates are for this particular input, which you can only see by looking at the probabilities. With temperature 0 greedy decoding: always the most likely token , I generated the full 40-token answer 20 times for each language. There was 1 distinct text per language, character for character. For the English CV it is "7" followed by the same reason every time, and for Japanese 「8点」. A fixed seed does the same for sampling: 10 runs with mx.random.seed 42 reset before each gave 1 distinct text. That is useful when you want variety but need to reproduce a particular run. It is not a fairness tool, though. With a seed, a different CV gets a different sequence of draws, so two candidates are still scored by different luck. One condition applies to all of this: every run here was a single request on an idle model. Servers that batch several requests together can produce tiny floating-point differences between runs, which only matter when two tokens are nearly tied. That was not measured here. Greedy decoding makes the output a function of the input. It doesn't make it a stable function. I made four edits that should not change a CV's score and took the greedy score and the probabilities each time: | Edit English | Greedy score | P top | Runner-up | |---|---|---|---| | none | 7 | 74.2% | 6 at 23.3% | | Skills line moved up | 7 | 69.6% | 6 at 28.1% | | Name changed Mia Chen → James Miller | 7 | 68.5% | 6 at 28.5% | | One extra blank line | 7 | 71.0% | 6 at 26.1% | | Question reworded | 6 | 67.5% | 5 at 19.3% | The reworded question was "Rate this CV for a junior data analyst role on a 1 to 10 scale. Give the score first, then one reason." Same task, same CV, and the greedy score drops from 7 to 6. In Japanese, all five versions stayed at 8 at temperature 0, but the top probability never went above 44%, and the runner-up switched between 9 and 7 depending on the edit. So temperature 0 removes the randomness between runs, not the sensitivity to the prompt. For comparing CVs this means: /completion endpoint returns token probabilities if you pass n probs , and with the pre-fill trick above the first token is the score. A 7 at 74% and an 8 at 40% are not the same kind of answer. In the osFoundry https://osfoundry.io desktop app, the same experiment is three panels with one local model in Models → Playground. Run the prompt once with the defaults to see the spread, then open the settings row and set Temp to 0. The temperature https://osfoundry.io/glossary/temperature and top-p https://osfoundry.io/glossary/top-p glossary entries cover the two settings in more detail. The app is free from the osFoundry download page https://osfoundry.io/download .