{"slug": "myth-faq-an-eval-score-is-not-a-measurement-and-six-reproducibility-traps", "title": "Myth FAQ: An Eval Score Is Not a Measurement (and Six Reproducibility Traps)", "summary": "A developer argues that a single LLM eval score is an anecdote with a decimal, not a measurement, and proposes reporting per-item flip rates across repeated runs instead. The writeup details six reproducibility traps — including temperature-zero variance, sampled LLM judges, prompt caching, and unhashed datasets — and offers an illustrative Python harness that runs each item several times and reports how often verdicts change.", "body_md": "Your eval score moved from 0.81 to 0.84. You shipped the change. Next week the old baseline won again.\n\nWas that improvement real? Or was it noise wearing a decimal point?\n\nI keep meeting the same pattern in agent work. A single number gets treated as a measurement. It is usually an anecdote with a decimal.\n\nHere are six myths I keep running into, plus a small harness that reports the number people actually need: how often an item flips.\n\nHosted inference is not a function. It is a service with batching, sampling, and version drift.\n\nTemperature zero reduces variance. It does not delete variance. Batching and backend changes still move tokens around.\n\nThe corrected mental model: every call is a sample. One call is a sample of size one.\n\nSo what does that make your score? A sample mean of an unknown distribution.\n\nMeans hide flips. Two runs can both score 0.80 and disagree on completely different items.\n\nAsk a sharper question instead. How many items changed verdict between runs?\n\nThat number is a flip rate. It tells you whether your delta is signal or jitter.\n\nA mean answers \"how did we do?\" A flip rate answers \"can I trust the diff?\" You need the second one to ship.\n\nIf an LLM grades your outputs, the grader is also a sampled system. Rubric wording moves it. Item order moves it. Context length moves it.\n\nSo measure the judge too. Feed the same saved outputs through it twice. If the judge flips, your score is a coin toss with extra steps.\n\nPrefer exact checks wherever possible:\n\n`jsonschema` for structured output.\nKeep the LLM judge for the cases nothing else can grade. Then report judge agreement next to the score.\n\nPrompt caching speeds things up. It also changes what the model actually sees.\n\nDeduplication shrinks your item set. Your denominator changes. Your score moves for free.\n\nRecord the item count and a hash of the input file in every report. If the hash changed, the comparison is not a comparison.\n\nFiles get edited. Items get added. Ordering changes. Someone quietly fixes a gold label.\n\nPin the dataset by hash, not by path. `items.jsonl` is a location. A SHA-256 is an identity.\n\nWhen a score jumps, check the dataset hash first. Then check the prompt hash. Then argue about the model.\n\nA 0.03 delta inside a 0.12 noise band is decoration, not progress.\n\nSet the gate from observed variance, not from hope. Run the harness a few times before you trust any threshold.\n\nAnd be honest about what you are comparing. A new prompt against an old score is a diff of two different weeks, not two different prompts.\n\nThe script below runs each item several times and reports per-item flip rate. It is illustrative, not a benchmark. Point it at your own items and endpoint.\n\n```\n# flip_rate.py - illustrative harness. Not a benchmark. Bring your own items.\nimport hashlib, json, os, statistics, urllib.request\n\nBASE  = os.environ['MODEL_BASE_URL'].rstrip('/')\nKEY   = os.environ['MODEL_API_KEY']\nMODEL = os.environ['MODEL_NAME']\nRUNS  = int(os.environ.get('RUNS', '5'))\nTEMP  = float(os.environ.get('TEMPERATURE', '0'))\n\ndef sha(text: str) -> str:\n    return hashlib.sha256(text.encode()).hexdigest()[:12]\n\ndef ask(item, seed):\n    body = json.dumps({\n        'model': MODEL,\n        'messages': item['messages'],\n        'temperature': TEMP,\n        'seed': seed,\n    }).encode()\n    req = urllib.request.Request(\n        BASE + '/chat/completions',\n        data=body,\n        headers={'Authorization': 'Bearer ' + KEY,\n                 'Content-Type': 'application/json'})\n    with urllib.request.urlopen(req, timeout=120) as r:\n        return json.load(r)['choices'][0]['message']['content']\n\ndef normalize(text: str) -> str:\n    # Keep the verdict, drop whitespace and case noise.\n    return ' '.join(text.strip().lower().split())\n\ndef flip_rate(outputs):\n    counts = {}\n    for o in outputs:\n        key = normalize(o)\n        counts[key] = counts.get(key, 0) + 1\n    return 1 - max(counts.values()) / len(outputs)\n\nitems = [json.loads(line) for line in open('items.jsonl')]\nreport, worst = [], 0.0\n\nfor item in items:\n    outs = [ask(item, seed=1000 + i) for i in range(RUNS)]\n    fr = flip_rate(outs)\n    worst = max(worst, fr)\n    report.append({\n        'id': item['id'],\n        'flip_rate': round(fr, 2),\n        'distinct_outputs': len({normalize(o) for o in outs}),\n        'sample_hash': sha(outs[0]),\n    })\n\nmanifest = {\n    'model': MODEL,\n    'temperature': TEMP,\n    'runs': RUNS,\n    'items_hash': sha(open('items.jsonl').read()),\n    'mean_flip_rate': round(\n        statistics.mean(r['flip_rate'] for r in report), 3),\n}\n\nprint(json.dumps({'manifest': manifest, 'items': report}, indent=2))\nraise SystemExit(1 if worst > float(os.environ.get('MAX_FLIP', '0.2')) else 0)\n```\n\nRun it like this in CI:\n\n```\npip install --quiet jsonschema\nRUNS=5 MAX_FLIP=0.2 python flip_rate.py > eval_report.json\n```\n\nTwo notes on the code. `seed` is a hint, not a contract; providers treat it differently, so verify in your own docs. And the script exits non-zero on a bad item, which makes it a gate instead of a dashboard nobody reads.\n\n| Flip rate | What it suggests | Reasonable action | \n|---|---|---|\n| 0.00 | Stable on this item set | Trust deltas above your noise floor | \n| 0.01 - 0.10 | Mild jitter | Require larger deltas before shipping | \n| 0.11 - 0.30 | Unstable items | Fix grading or tighten the prompt | \n| Above 0.30 | Coin toss | Stop tuning and repair the harness | \n\nThese bands are a starting heuristic, not measured constants. Calibrate them on your own set before you enforce anything.\n\nThis harness is repetitive by design. It wants several calls per item and a place to run them.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nThe operator describes free model access and a free server option. Those two things are genuinely useful here, because flip-rate measurement burns calls and wall-clock time for no user-visible feature.\n\nA free endpoint lets you take the first pass across more items. A free server lets the loop run somewhere other than your laptop lid. I cannot speak to quotas, limits, or longevity, and you should check current terms before you build a pipeline on it.\n\nPoint the harness at whatever endpoint you have. The script does not care who answers.\n\nBe clear about what this does not do.\n\n`429`, stop and back off. Do not wrap the harness in a hot retry loop.\nSkip this approach if you have fewer than about ten items. Skip it for smoke tests where any answer is acceptable. Skip it for one-shot creative tasks with no ground truth to compare.\n\nUse it when a prompt change needs a decision, and the only evidence is a number that keeps wobbling.\n\nBefore you cite the next score, ask one thing. If I re-ran this today, how many items would change?\n\nIf you cannot answer that, you have a decimal point, not a measurement.", "url": "https://wpnews.pro/news/myth-faq-an-eval-score-is-not-a-measurement-and-six-reproducibility-traps", "canonical_source": "https://dev.to/gitlab_3188/myth-faq-an-eval-score-is-not-a-measurement-and-six-reproducibility-traps-5cf0", "published_at": "2026-10-16 19:43:27+00:00", "updated_at": "2026-10-16 19:50:32.784251+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "mlops", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/myth-faq-an-eval-score-is-not-a-measurement-and-six-reproducibility-traps", "markdown": "https://wpnews.pro/news/myth-faq-an-eval-score-is-not-a-measurement-and-six-reproducibility-traps.md", "text": "https://wpnews.pro/news/myth-faq-an-eval-score-is-not-a-measurement-and-six-reproducibility-traps.txt", "jsonld": "https://wpnews.pro/news/myth-faq-an-eval-score-is-not-a-measurement-and-six-reproducibility-traps.jsonld"}}