Why does a local LLM give a different score every time, and does temperature 0 fix it?

Give a language model a CV and ask for a score from 1 to 10, and you get a number and a confident reason. Ask again and the number can move. The usual advice is "set temperature to 0", which is correct but leaves two questions open: how much does the score actually move with default settings, and what does temperature 0 not fix? Both can be measured on a laptop in a few minutes, because the score is a single token and the model hands you its probability directly.

The setup below uses Qwen2.5-3B-Instruct (4-bit, MLX) with mlx-lm 0.32.2 on an 8 GB M1 Mac. The prompts are the ones from a CV-scoring demo in the osFoundry desktop app's Models → Playground: a six-line CV for a junior data analyst role, followed by "Score this CV from 1 to 10 for a junior data analyst role. Start with the score, then one reason." The app runs local models through llama.cpp's llama-server, so the sampler settings come from there.

When a request carries no temperature, llama-server samples with its own defaults. The build bundled with the app (version 10358) lists them in --help:

--temp, --temperature N    temperature (default: 0.80)
--top-k N                  top-k sampling (default: 40, 0 = disabled)
--top-p N                  top-p sampling (default: 0.95, 1.0 = disabled)
--min-p N                  min-p sampling (default: 0.05, 0.0 = disabled)
-s, --seed SEED            RNG seed (default: -1, use random seed for -1)

These apply to every request that doesn't set its own values. When the app starts llama-server, it only adds --temp if a temperature has been configured for the server (local_inference.rs), so a request without a temperature samples at 0.8 with a random seed, which is also what most llama.cpp front ends do out of the box.

A free-form answer can start with "Score: 7", "7/10" or "I would rate this CV a 7", which makes the score hard to locate. The fix is to pre-fill the start of the reply, so the very next token the model produces is the digit:

import mlx.core as mx
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tok = load("mlx-community/Qwen2.5-3B-Instruct-4bit")
DIGITS = {d: tok.encode(str(d))[0] for d in range(1, 10)}

def ids_for(cv, question, prefill="Score: "):
    msgs = [{"role": "user", "content": cv + "\n\n" + question}]
    text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
    return tok.encode(text + prefill)

def digit_probs(ids, T=1.0):
    logits = model(mx.array([ids]))[0, -1].astype(mx.float32)
    p = mx.softmax(logits / T)
    return {d: float(p[t]) for d, t in DIGITS.items()}

Qwen's tokenizer splits numbers into single digits, so "7" is one token and "10" would start with the token "1". One forward pass over the 173-token prompt gives the whole distribution of the score.

For the English CV, at temperature 1 (the model's raw probabilities):

The model doesn't "think" the CV is a 7. It thinks it is probably a 7, with a one-in-four chance of a 6. Temperature divides the logits before the softmax, which sharpens or flattens this:

llama-server doesn't apply temperature alone. Its default chain runs top-k 40, then top-p 0.95, then min-p 0.05, and temperature last. Applied to these logits, top-p already cuts everything after 7 and 6 (together they are over 95%), so the default sampler picks between exactly two scores:

def llamacpp_default(logits, temp=0.8, top_k=40, top_p=0.95, min_p=0.05):
    order = mx.argsort(-logits)[:top_k]
    kept = logits[order]
    p = mx.softmax(kept)
    n = int(mx.sum(mx.cumsum(p) < top_p).item()) + 1
    idx = [i for i in range(n) if p[i].item() >= min_p * p[0].item()]
    q = mx.softmax(kept[mx.array(idx)] / temp)
    return {tok.decode([int(order[i])]): float(q[j]) for j, i in enumerate(idx)}

Then I sampled it for real: 200 generations from the same prompt with mlx-lm's sampler set to the same four values (make_sampler(temp=0.8, top_p=0.95, min_p=0.05, top_k=40)).

So with default settings, about one run in five scores this CV a 6 instead of a 7. If you score two candidates once each and one gets a 6 and the other a 7, there is a real chance the gap is the draw.

The same CV and question in Japanese (222 tokens, pre-fill 「点数：」) gave a much less decided model:

Here the most likely score wins only about half the time, and four different scores came out of 200 runs. How much a score wobbles is not a fixed property of the model. It depends on how close the top candidates are for this particular input, which you can only see by looking at the probabilities.

With temperature 0 (greedy decoding: always the most likely token), I generated the full 40-token answer 20 times for each language. There was 1 distinct text per language, character for character. For the English CV it is "7" followed by the same reason every time, and for Japanese 「8点」.

A fixed seed does the same for sampling: 10 runs with mx.random.seed(42) reset before each gave 1 distinct text. That is useful when you want variety but need to reproduce a particular run. It is not a fairness tool, though. With a seed, a different CV gets a different sequence of draws, so two candidates are still scored by different luck.

One condition applies to all of this: every run here was a single request on an idle model. Servers that batch several requests together can produce tiny floating-point differences between runs, which only matter when two tokens are nearly tied. That was not measured here.

Greedy decoding makes the output a function of the input. It doesn't make it a stable function. I made four edits that should not change a CV's score and took the greedy score and the probabilities each time:

The reworded question was "Rate this CV for a junior data analyst role on a 1 to 10 scale. Give the score first, then one reason." Same task, same CV, and the greedy score drops from 7 to 6. In Japanese, all five versions stayed at 8 at temperature 0, but the top probability never went above 44%, and the runner-up switched between 9 and 7 depending on the edit.

So temperature 0 removes the randomness between runs, not the sensitivity to the prompt. For comparing CVs this means:

/completion endpoint returns token probabilities if you pass n_probs, and with the pre-fill trick above the first token is the score. A 7 at 74% and an 8 at 40% are not the same kind of answer. In the osFoundry desktop app, the same experiment is three panels with one local model in Models → Playground. Run the prompt once with the defaults to see the spread, then open the settings row and set Temp to 0.

The temperature and top-p glossary entries cover the two settings in more detail. The app is free from the osFoundry download page.