# Why does a local LLM give a different score every time, and does temperature 0 fix it?

> Source: <https://dev.to/bestaitool/why-does-a-local-llm-give-a-different-score-every-time-and-does-temperature-0-fix-it-8o>
> Published: 2026-11-09 06:46:46+00:00

Give a language model a CV and ask for a score from 1 to 10, and you get a number and a confident reason. Ask again and the number can move. The usual advice is "set temperature to 0", which is correct but leaves two questions open: how much does the score actually move with default settings, and what does temperature 0 not fix? Both can be measured on a laptop in a few minutes, because the score is a single token and the model hands you its probability directly.

The setup below uses Qwen2.5-3B-Instruct (4-bit, MLX) with mlx-lm 0.32.2 on an 8 GB M1 Mac. The prompts are the ones from a CV-scoring demo in the osFoundry desktop app's Models → Playground: a six-line CV for a junior data analyst role, followed by "Score this CV from 1 to 10 for a junior data analyst role. Start with the score, then one reason." The app runs local models through llama.cpp's `llama-server`, so the sampler settings come from there.

When a request carries no temperature, llama-server samples with its own defaults. The build bundled with the app (version 10358) lists them in `--help`:

```
--temp, --temperature N    temperature (default: 0.80)
--top-k N                  top-k sampling (default: 40, 0 = disabled)
--top-p N                  top-p sampling (default: 0.95, 1.0 = disabled)
--min-p N                  min-p sampling (default: 0.05, 0.0 = disabled)
-s, --seed SEED            RNG seed (default: -1, use random seed for -1)
```

These apply to every request that doesn't set its own values. When the app starts llama-server, it only adds `--temp` if a temperature has been configured for the server (`local_inference.rs`), so a request without a temperature samples at 0.8 with a random seed, which is also what most llama.cpp front ends do out of the box.

A free-form answer can start with "Score: 7", "**7/10**" or "I would rate this CV a 7", which makes the score hard to locate. The fix is to pre-fill the start of the reply, so the very next token the model produces is the digit:

``` python
import mlx.core as mx
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("mlx-community/Qwen2.5-3B-Instruct-4bit")
DIGITS = {d: tok.encode(str(d))[0] for d in range(1, 10)}

def ids_for(cv, question, prefill="Score: "):
    msgs = [{"role": "user", "content": cv + "\n\n" + question}]
    text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
    return tok.encode(text + prefill)

def digit_probs(ids, T=1.0):
    logits = model(mx.array([ids]))[0, -1].astype(mx.float32)
    p = mx.softmax(logits / T)
    return {d: float(p[t]) for d, t in DIGITS.items()}
```

Qwen's tokenizer splits numbers into single digits, so "7" is one token and "10" would start with the token "1". One forward pass over the 173-token prompt gives the whole distribution of the score.

For the English CV, at temperature 1 (the model's raw probabilities):

| Score | Probability | 
|---|---|
| 7 | 74.2% | 
| 6 | 23.3% | 
| 8 | 1.3% | 
| 5 | 1.2% | 

The model doesn't "think" the CV is a 7. It thinks it is probably a 7, with a one-in-four chance of a 6. Temperature divides the logits before the softmax, which sharpens or flattens this:

| Temperature | P(7) | P(6) | P(8) | 
|---|---|---|---|
| 1.0 | 74.2% | 23.3% | 1.3% | 
| 0.8 | 80.2% | 18.9% | 0.5% | 
| 0.5 | 91.0% | 9.0% | 0.0% | 
| 0.2 | 99.7% | 0.3% | 0.0% | 

llama-server doesn't apply temperature alone. Its default chain runs top-k 40, then top-p 0.95, then min-p 0.05, and temperature last. Applied to these logits, top-p already cuts everything after 7 and 6 (together they are over 95%), so the default sampler picks between exactly two scores:

``` python
def llamacpp_default(logits, temp=0.8, top_k=40, top_p=0.95, min_p=0.05):
    order = mx.argsort(-logits)[:top_k]
    kept = logits[order]
    p = mx.softmax(kept)
    n = int(mx.sum(mx.cumsum(p) < top_p).item()) + 1
    idx = [i for i in range(n) if p[i].item() >= min_p * p[0].item()]
    q = mx.softmax(kept[mx.array(idx)] / temp)
    return {tok.decode([int(order[i])]): float(q[j]) for j, i in enumerate(idx)}
# -> {'7': 0.8093, '6': 0.1907}
```

Then I sampled it for real: 200 generations from the same prompt with mlx-lm's sampler set to the same four values (`make_sampler(temp=0.8, top_p=0.95, min_p=0.05, top_k=40)`).

|  | 7 | 6 | 
|---|---|---|
| Predicted by the chain | 80.9% | 19.1% | 
| Observed, 200 runs | 162 (81%) | 38 (19%) | 

So with default settings, about one run in five scores this CV a 6 instead of a 7. If you score two candidates once each and one gets a 6 and the other a 7, there is a real chance the gap is the draw.

The same CV and question in Japanese (222 tokens, pre-fill 「点数：」) gave a much less decided model:

| Score | Probability at T=1 | Observed, 200 runs at the defaults | 
|---|---|---|
| 8 | 39.8% | 108 | 
| 9 | 23.8% | 44 | 
| 7 | 19.4% | 40 | 
| 10 | (starts with "1": 4.1%) | 8 | 

Here the most likely score wins only about half the time, and four different scores came out of 200 runs. How much a score wobbles is not a fixed property of the model. It depends on how close the top candidates are for this particular input, which you can only see by looking at the probabilities.

With temperature 0 (greedy decoding: always the most likely token), I generated the full 40-token answer 20 times for each language. There was 1 distinct text per language, character for character. For the English CV it is "7" followed by the same reason every time, and for Japanese 「8点」.

A fixed seed does the same for sampling: 10 runs with `mx.random.seed(42)` reset before each gave 1 distinct text. That is useful when you want variety but need to reproduce a particular run. It is not a fairness tool, though. With a seed, a different CV gets a different sequence of draws, so two candidates are still scored by different luck.

One condition applies to all of this: every run here was a single request on an idle model. Servers that batch several requests together can produce tiny floating-point differences between runs, which only matter when two tokens are nearly tied. That was not measured here.

Greedy decoding makes the output a function of the input. It doesn't make it a stable function. I made four edits that should not change a CV's score and took the greedy score and the probabilities each time:

| Edit (English) | Greedy score | P(top) | Runner-up | 
|---|---|---|---|
| none | 7 | 74.2% | 6 at 23.3% | 
| Skills line moved up | 7 | 69.6% | 6 at 28.1% | 
| Name changed (Mia Chen → James Miller) | 7 | 68.5% | 6 at 28.5% | 
| One extra blank line | 7 | 71.0% | 6 at 26.1% | 
| Question reworded | 6 | 67.5% | 5 at 19.3% | 

The reworded question was "Rate this CV for a junior data analyst role on a 1 to 10 scale. Give the score first, then one reason." Same task, same CV, and the greedy score drops from 7 to 6. In Japanese, all five versions stayed at 8 at temperature 0, but the top probability never went above 44%, and the runner-up switched between 9 and 7 depending on the edit.

So temperature 0 removes the randomness between runs, not the sensitivity to the prompt. For comparing CVs this means:

`/completion` endpoint returns token probabilities if you pass `n_probs`, and with the pre-fill trick above the first token is the score. A 7 at 74% and an 8 at 40% are not the same kind of answer.
In the [osFoundry](https://osfoundry.io) desktop app, the same experiment is three panels with one local model in Models → Playground. Run the prompt once with the defaults to see the spread, then open the settings row and set Temp to 0.

The [temperature](https://osfoundry.io/glossary/temperature) and [top-p](https://osfoundry.io/glossary/top-p) glossary entries cover the two settings in more detail. The app is free from the [osFoundry download page](https://osfoundry.io/download).