{"slug": "why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-it", "title": "Why does a local LLM give a different score every time, and does temperature 0 fix it?", "summary": "A developer measured how much a local LLM's CV-scoring output varies by reading the model's raw next-token probabilities, using Qwen2.5-3B-Instruct (4-bit, MLX) on an 8 GB M1 Mac. A single forward pass over a pre-filled \"Score: \" prompt showed the model assigns 74.2% probability to a 7 and 23.3% to a 6 at temperature 1, and that llama.cpp's default sampler chain (top-k 40, top-p 0.95, min-p 0.05, temp 0.8) narrows the choice to just those two scores. The work also shows what temperature 0 does not fix, since the underlying distribution remains probabilistic.", "body_md": "Give a language model a CV and ask for a score from 1 to 10, and you get a number and a confident reason. Ask again and the number can move. The usual advice is \"set temperature to 0\", which is correct but leaves two questions open: how much does the score actually move with default settings, and what does temperature 0 not fix? Both can be measured on a laptop in a few minutes, because the score is a single token and the model hands you its probability directly.\n\nThe setup below uses Qwen2.5-3B-Instruct (4-bit, MLX) with mlx-lm 0.32.2 on an 8 GB M1 Mac. The prompts are the ones from a CV-scoring demo in the osFoundry desktop app's Models → Playground: a six-line CV for a junior data analyst role, followed by \"Score this CV from 1 to 10 for a junior data analyst role. Start with the score, then one reason.\" The app runs local models through llama.cpp's `llama-server`, so the sampler settings come from there.\n\nWhen a request carries no temperature, llama-server samples with its own defaults. The build bundled with the app (version 10358) lists them in `--help`:\n\n```\n--temp, --temperature N    temperature (default: 0.80)\n--top-k N                  top-k sampling (default: 40, 0 = disabled)\n--top-p N                  top-p sampling (default: 0.95, 1.0 = disabled)\n--min-p N                  min-p sampling (default: 0.05, 0.0 = disabled)\n-s, --seed SEED            RNG seed (default: -1, use random seed for -1)\n```\n\nThese apply to every request that doesn't set its own values. When the app starts llama-server, it only adds `--temp` if a temperature has been configured for the server (`local_inference.rs`), so a request without a temperature samples at 0.8 with a random seed, which is also what most llama.cpp front ends do out of the box.\n\nA free-form answer can start with \"Score: 7\", \"**7/10**\" or \"I would rate this CV a 7\", which makes the score hard to locate. The fix is to pre-fill the start of the reply, so the very next token the model produces is the digit:\n\n``` python\nimport mlx.core as mx\nfrom mlx_lm import load, generate\nfrom mlx_lm.sample_utils import make_sampler\n\nmodel, tok = load(\"mlx-community/Qwen2.5-3B-Instruct-4bit\")\nDIGITS = {d: tok.encode(str(d))[0] for d in range(1, 10)}\n\ndef ids_for(cv, question, prefill=\"Score: \"):\n    msgs = [{\"role\": \"user\", \"content\": cv + \"\\n\\n\" + question}]\n    text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)\n    return tok.encode(text + prefill)\n\ndef digit_probs(ids, T=1.0):\n    logits = model(mx.array([ids]))[0, -1].astype(mx.float32)\n    p = mx.softmax(logits / T)\n    return {d: float(p[t]) for d, t in DIGITS.items()}\n```\n\nQwen's tokenizer splits numbers into single digits, so \"7\" is one token and \"10\" would start with the token \"1\". One forward pass over the 173-token prompt gives the whole distribution of the score.\n\nFor the English CV, at temperature 1 (the model's raw probabilities):\n\n| Score | Probability | \n|---|---|\n| 7 | 74.2% | \n| 6 | 23.3% | \n| 8 | 1.3% | \n| 5 | 1.2% | \n\nThe model doesn't \"think\" the CV is a 7. It thinks it is probably a 7, with a one-in-four chance of a 6. Temperature divides the logits before the softmax, which sharpens or flattens this:\n\n| Temperature | P(7) | P(6) | P(8) | \n|---|---|---|---|\n| 1.0 | 74.2% | 23.3% | 1.3% | \n| 0.8 | 80.2% | 18.9% | 0.5% | \n| 0.5 | 91.0% | 9.0% | 0.0% | \n| 0.2 | 99.7% | 0.3% | 0.0% | \n\nllama-server doesn't apply temperature alone. Its default chain runs top-k 40, then top-p 0.95, then min-p 0.05, and temperature last. Applied to these logits, top-p already cuts everything after 7 and 6 (together they are over 95%), so the default sampler picks between exactly two scores:\n\n``` python\ndef llamacpp_default(logits, temp=0.8, top_k=40, top_p=0.95, min_p=0.05):\n    order = mx.argsort(-logits)[:top_k]\n    kept = logits[order]\n    p = mx.softmax(kept)\n    n = int(mx.sum(mx.cumsum(p) < top_p).item()) + 1\n    idx = [i for i in range(n) if p[i].item() >= min_p * p[0].item()]\n    q = mx.softmax(kept[mx.array(idx)] / temp)\n    return {tok.decode([int(order[i])]): float(q[j]) for j, i in enumerate(idx)}\n# -> {'7': 0.8093, '6': 0.1907}\n```\n\nThen I sampled it for real: 200 generations from the same prompt with mlx-lm's sampler set to the same four values (`make_sampler(temp=0.8, top_p=0.95, min_p=0.05, top_k=40)`).\n\n|  | 7 | 6 | \n|---|---|---|\n| Predicted by the chain | 80.9% | 19.1% | \n| Observed, 200 runs | 162 (81%) | 38 (19%) | \n\nSo with default settings, about one run in five scores this CV a 6 instead of a 7. If you score two candidates once each and one gets a 6 and the other a 7, there is a real chance the gap is the draw.\n\nThe same CV and question in Japanese (222 tokens, pre-fill 「点数：」) gave a much less decided model:\n\n| Score | Probability at T=1 | Observed, 200 runs at the defaults | \n|---|---|---|\n| 8 | 39.8% | 108 | \n| 9 | 23.8% | 44 | \n| 7 | 19.4% | 40 | \n| 10 | (starts with \"1\": 4.1%) | 8 | \n\nHere the most likely score wins only about half the time, and four different scores came out of 200 runs. How much a score wobbles is not a fixed property of the model. It depends on how close the top candidates are for this particular input, which you can only see by looking at the probabilities.\n\nWith temperature 0 (greedy decoding: always the most likely token), I generated the full 40-token answer 20 times for each language. There was 1 distinct text per language, character for character. For the English CV it is \"7\" followed by the same reason every time, and for Japanese 「8点」.\n\nA fixed seed does the same for sampling: 10 runs with `mx.random.seed(42)` reset before each gave 1 distinct text. That is useful when you want variety but need to reproduce a particular run. It is not a fairness tool, though. With a seed, a different CV gets a different sequence of draws, so two candidates are still scored by different luck.\n\nOne condition applies to all of this: every run here was a single request on an idle model. Servers that batch several requests together can produce tiny floating-point differences between runs, which only matter when two tokens are nearly tied. That was not measured here.\n\nGreedy decoding makes the output a function of the input. It doesn't make it a stable function. I made four edits that should not change a CV's score and took the greedy score and the probabilities each time:\n\n| Edit (English) | Greedy score | P(top) | Runner-up | \n|---|---|---|---|\n| none | 7 | 74.2% | 6 at 23.3% | \n| Skills line moved up | 7 | 69.6% | 6 at 28.1% | \n| Name changed (Mia Chen → James Miller) | 7 | 68.5% | 6 at 28.5% | \n| One extra blank line | 7 | 71.0% | 6 at 26.1% | \n| Question reworded | 6 | 67.5% | 5 at 19.3% | \n\nThe reworded question was \"Rate this CV for a junior data analyst role on a 1 to 10 scale. Give the score first, then one reason.\" Same task, same CV, and the greedy score drops from 7 to 6. In Japanese, all five versions stayed at 8 at temperature 0, but the top probability never went above 44%, and the runner-up switched between 9 and 7 depending on the edit.\n\nSo temperature 0 removes the randomness between runs, not the sensitivity to the prompt. For comparing CVs this means:\n\n`/completion` endpoint returns token probabilities if you pass `n_probs`, and with the pre-fill trick above the first token is the score. A 7 at 74% and an 8 at 40% are not the same kind of answer.\nIn the [osFoundry](https://osfoundry.io) desktop app, the same experiment is three panels with one local model in Models → Playground. Run the prompt once with the defaults to see the spread, then open the settings row and set Temp to 0.\n\nThe [temperature](https://osfoundry.io/glossary/temperature) and [top-p](https://osfoundry.io/glossary/top-p) glossary entries cover the two settings in more detail. The app is free from the [osFoundry download page](https://osfoundry.io/download).", "url": "https://wpnews.pro/news/why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-it", "canonical_source": "https://dev.to/bestaitool/why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-fix-it-8o", "published_at": "2026-11-09 06:46:46+00:00", "updated_at": "2026-11-09 06:52:20.892201+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-research"], "entities": ["Qwen2.5-3B-Instruct", "MLX", "mlx-lm", "llama.cpp", "llama-server", "osFoundry"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-it", "markdown": "https://wpnews.pro/news/why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-it.md", "text": "https://wpnews.pro/news/why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-it.txt", "jsonld": "https://wpnews.pro/news/why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-it.jsonld"}}