Myth FAQ: An Eval Score Is Not a Measurement (and Six Reproducibility Traps)

A developer argues that a single LLM eval score is an anecdote with a decimal, not a measurement, and proposes reporting per-item flip rates across repeated runs instead. The writeup details six reproducibility traps — including temperature-zero variance, sampled LLM judges, prompt caching, and unhashed datasets — and offers an illustrative Python harness that runs each item several times and reports how often verdicts change.

Your eval score moved from 0.81 to 0.84. You shipped the change. Next week the old baseline won again. Was that improvement real? Or was it noise wearing a decimal point? I keep meeting the same pattern in agent work. A single number gets treated as a measurement. It is usually an anecdote with a decimal. Here are six myths I keep running into, plus a small harness that reports the number people actually need: how often an item flips. Hosted inference is not a function. It is a service with batching, sampling, and version drift. Temperature zero reduces variance. It does not delete variance. Batching and backend changes still move tokens around. The corrected mental model: every call is a sample. One call is a sample of size one. So what does that make your score? A sample mean of an unknown distribution. Means hide flips. Two runs can both score 0.80 and disagree on completely different items. Ask a sharper question instead. How many items changed verdict between runs? That number is a flip rate. It tells you whether your delta is signal or jitter. A mean answers "how did we do?" A flip rate answers "can I trust the diff?" You need the second one to ship. If an LLM grades your outputs, the grader is also a sampled system. Rubric wording moves it. Item order moves it. Context length moves it. So measure the judge too. Feed the same saved outputs through it twice. If the judge flips, your score is a coin toss with extra steps. Prefer exact checks wherever possible: jsonschema for structured output. Keep the LLM judge for the cases nothing else can grade. Then report judge agreement next to the score. Prompt caching speeds things up. It also changes what the model actually sees. Deduplication shrinks your item set. Your denominator changes. Your score moves for free. Record the item count and a hash of the input file in every report. If the hash changed, the comparison is not a comparison. Files get edited. Items get added. Ordering changes. Someone quietly fixes a gold label. Pin the dataset by hash, not by path. items.jsonl is a location. A SHA-256 is an identity. When a score jumps, check the dataset hash first. Then check the prompt hash. Then argue about the model. A 0.03 delta inside a 0.12 noise band is decoration, not progress. Set the gate from observed variance, not from hope. Run the harness a few times before you trust any threshold. And be honest about what you are comparing. A new prompt against an old score is a diff of two different weeks, not two different prompts. The script below runs each item several times and reports per-item flip rate. It is illustrative, not a benchmark. Point it at your own items and endpoint. flip rate.py - illustrative harness. Not a benchmark. Bring your own items. import hashlib, json, os, statistics, urllib.request BASE = os.environ 'MODEL BASE URL' .rstrip '/' KEY = os.environ 'MODEL API KEY' MODEL = os.environ 'MODEL NAME' RUNS = int os.environ.get 'RUNS', '5' TEMP = float os.environ.get 'TEMPERATURE', '0' def sha text: str - str: return hashlib.sha256 text.encode .hexdigest :12 def ask item, seed : body = json.dumps { 'model': MODEL, 'messages': item 'messages' , 'temperature': TEMP, 'seed': seed, } .encode req = urllib.request.Request BASE + '/chat/completions', data=body, headers={'Authorization': 'Bearer ' + KEY, 'Content-Type': 'application/json'} with urllib.request.urlopen req, timeout=120 as r: return json.load r 'choices' 0 'message' 'content' def normalize text: str - str: Keep the verdict, drop whitespace and case noise. return ' '.join text.strip .lower .split def flip rate outputs : counts = {} for o in outputs: key = normalize o counts key = counts.get key, 0 + 1 return 1 - max counts.values / len outputs items = json.loads line for line in open 'items.jsonl' report, worst = , 0.0 for item in items: outs = ask item, seed=1000 + i for i in range RUNS fr = flip rate outs worst = max worst, fr report.append { 'id': item 'id' , 'flip rate': round fr, 2 , 'distinct outputs': len {normalize o for o in outs} , 'sample hash': sha outs 0 , } manifest = { 'model': MODEL, 'temperature': TEMP, 'runs': RUNS, 'items hash': sha open 'items.jsonl' .read , 'mean flip rate': round statistics.mean r 'flip rate' for r in report , 3 , } print json.dumps {'manifest': manifest, 'items': report}, indent=2 raise SystemExit 1 if worst float os.environ.get 'MAX FLIP', '0.2' else 0 Run it like this in CI: pip install --quiet jsonschema RUNS=5 MAX FLIP=0.2 python flip rate.py eval report.json Two notes on the code. seed is a hint, not a contract; providers treat it differently, so verify in your own docs. And the script exits non-zero on a bad item, which makes it a gate instead of a dashboard nobody reads. | Flip rate | What it suggests | Reasonable action | |---|---|---| | 0.00 | Stable on this item set | Trust deltas above your noise floor | | 0.01 - 0.10 | Mild jitter | Require larger deltas before shipping | | 0.11 - 0.30 | Unstable items | Fix grading or tighten the prompt | | Above 0.30 | Coin toss | Stop tuning and repair the harness | These bands are a starting heuristic, not measured constants. Calibrate them on your own set before you enforce anything. This harness is repetitive by design. It wants several calls per item and a place to run them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes free model access and a free server option. Those two things are genuinely useful here, because flip-rate measurement burns calls and wall-clock time for no user-visible feature. A free endpoint lets you take the first pass across more items. A free server lets the loop run somewhere other than your laptop lid. I cannot speak to quotas, limits, or longevity, and you should check current terms before you build a pipeline on it. Point the harness at whatever endpoint you have. The script does not care who answers. Be clear about what this does not do. 429 , stop and back off. Do not wrap the harness in a hot retry loop. Skip this approach if you have fewer than about ten items. Skip it for smoke tests where any answer is acceptable. Skip it for one-shot creative tasks with no ground truth to compare. Use it when a prompt change needs a decision, and the only evidence is a number that keeps wobbling. Before you cite the next score, ask one thing. If I re-ran this today, how many items would change? If you cannot answer that, you have a decimal point, not a measurement.