{"slug": "your-llm-as-judge-disagrees-with-itself-between-runs", "title": "Your LLM-as-judge disagrees with itself between runs", "summary": "An engineer at an unnamed company discovered that their LLM-as-judge gate for evaluating faithfulness was unreliable, producing different scores on identical inputs due to factors like sampling temperature, model version drift, prompt ambiguity, and tie-breaking. To fix the flapping gate, they implemented temperature 0, seeded generation, pinned model versions, averaged over multiple samples, quantized scores, and version-controlled the judge prompt. The key insight is to treat a single judged score as a sample from a distribution and only flag regressions when the mean falls below the threshold by more than the measured noise.", "body_md": "Same outputs, same judge, two runs, two scores. The gate flickered red then green on a branch with zero code changes, and that flapping cost me more trust than any real regression.\n\nI had a faithfulness gate on merge: judge scores every case, the mean has to clear 0.80. One Tuesday it failed at 0.79. I re-ran the identical job, no code change, no prompt change, and it passed at 0.82. Ran it a third time: 0.80 exactly. Nothing in the repo had moved. The judge was disagreeing with itself.\n\nA gate that returns a different verdict on the same inputs is worse than no gate. People stop believing the red, they re-run until it goes green, and now the check is a slot machine you pull until it pays out. The regression it was supposed to catch could sail through on the lucky pull. So before I trusted that gate again I had to make the judge reproducible enough to stand on.\n\nFour sources, in the order they bit me.\n\nSampling temperature. A judge call is a generation. If temperature is above zero the model samples, and a borderline case lands on 4/5 one time and 3/5 the next. This is the biggest lever and the easiest to miss because most SDK defaults are not zero.\n\nModel version drift. \"gpt-4o\" or \"claude-latest\" is a moving alias. The provider ships a new snapshot, your scores shift a few points overnight, and you blame your prompt. Pin the dated snapshot, not the floating name.\n\nPrompt ambiguity. If your rubric says \"rate helpfulness 1 to 5\" without anchoring what a 3 versus a 4 means, the model resolves the ambiguity differently each call. Vague rubrics convert directly into variance.\n\nTie-breaking. When the judge is genuinely on the fence between two scores, tiny sampling noise decides, and that decision is exactly where your threshold tends to sit.\n\nYou will not get bit-identical determinism from a hosted model. That is fine. The goal is not zero noise, it is noise small enough that a threshold crossing means a real change and not a coin flip. Five things got me there.\n\nTemperature 0, and a seed where the provider supports one. This alone collapsed most of my flap. Seeds help further on providers that honor them, but do not assume a seed gives you exact reproducibility across a model update.\n\nPin the exact judge model and prompt version in the cache key. Same discipline as any eval cache: the score is only reusable if the input, the judge snapshot, and the rubric version all match. Bump the version string whenever you touch the rubric.\n\nAverage over k judged samples, or take majority vote. One call is a sample from a distribution. k calls and a mean (or a vote for pass/fail rubrics) shrink the variance of your estimate by roughly sqrt(k). I run k=5.\n\nQuantize the score. If you gate on a continuous 0 to 1, every hundredth flaps. Round to a coarse grid (0.0, 0.25, 0.5, 0.75, 1.0) per case so sub-grid noise stops moving the aggregate.\n\nVersion the judge prompt as code. The rubric lives in the repo, gets a version string, and changes go through review. A judge prompt edited in a UI and not tracked is a silent score change you cannot bisect.\n\nThe real fix is conceptual: stop treating one judged score as ground truth. Judge k times, keep the mean and the spread, and only fail when the mean is below the threshold by more than the noise you actually measured. If the mean sits inside the noise band around the threshold, that is not a regression, it is jitter, and failing on it is how you get a flapping gate.\n\n``` python\nimport statistics\nfrom typing import Callable\n\ndef stable_judge_score(\n    judge: Callable[[str, str], float],\n    output: str,\n    reference: str,\n    k: int = 5,\n    quantize_to: float = 0.25,\n) -> tuple[float, float]:\n    \"\"\"Run the judge k times at temperature 0. Return (mean, stdev),\n    each raw score snapped to a coarse grid to kill sub-grid jitter.\"\"\"\n    scores = []\n    for _ in range(k):\n        raw = judge(output, reference)          # judge must be called at temperature 0\n        snapped = round(raw / quantize_to) * quantize_to\n        scores.append(snapped)\n    mean = statistics.fmean(scores)\n    stdev = statistics.pstdev(scores) if k > 1 else 0.0\n    return mean, stdev\n\ndef gate(mean: float, stdev: float, threshold: float = 0.80) -> bool:\n    \"\"\"Fail only when the mean is below threshold by more than the\n    observed noise. Inside the noise band counts as pass, not a flap.\"\"\"\n    return mean >= (threshold - stdev)\n\nif __name__ == \"__main__\":\n    # toy judge: deterministic here, real one hits an LLM at temperature 0\n    def fake_judge(out: str, ref: str) -> float:\n        return 0.79 if \"borderline\" in out else 0.9\n\n    mean, stdev = stable_judge_score(fake_judge, \"a borderline answer\", \"ref\", k=5)\n    passed = gate(mean, stdev, threshold=0.80)\n    print(f\"mean={mean:.3f} stdev={stdev:.3f} pass={passed}\")\n    raise SystemExit(0 if passed else 1)   # this exit code is what CI reads\n```\n\nThe raise SystemExit is the load-bearing line. That is what makes the branch protection rule refuse a real regression. Everything above it exists so that exit code means something. On my suite, moving from one raw judge call to k=5 with quantization took the run-to-run swing on that faithfulness metric from about 0.03 down to under 0.01, which was finally tight enough that a red meant a real drop and people stopped re-running to dodge it.\n\nOne caution on k: more samples cost more judge calls and more wall-clock, so I only spend the k on the cases near the threshold, and run the obviously-passing and obviously-failing cases once. The noise only matters where the decision is close.", "url": "https://wpnews.pro/news/your-llm-as-judge-disagrees-with-itself-between-runs", "canonical_source": "https://dev.to/ethanwritesai/your-llm-as-judge-disagrees-with-itself-between-runs-1e3e", "published_at": "2026-07-08 19:51:30+00:00", "updated_at": "2026-07-08 20:11:24.734354+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/your-llm-as-judge-disagrees-with-itself-between-runs", "markdown": "https://wpnews.pro/news/your-llm-as-judge-disagrees-with-itself-between-runs.md", "text": "https://wpnews.pro/news/your-llm-as-judge-disagrees-with-itself-between-runs.txt", "jsonld": "https://wpnews.pro/news/your-llm-as-judge-disagrees-with-itself-between-runs.jsonld"}}