{"slug": "the-two-sources-of-noise-every-llm-evaluation-system-must-handle", "title": "The Two Sources of Noise Every LLM Evaluation System Must Handle", "summary": "LLM evaluations suffer from two sources of noise: ambiguous inputs and inconsistent judges, according to a technical analysis. The author's team found that treating evaluation scores as static measurements rather than estimates leads to credibility loss, and recommends fixing inputs by using multi-turn conversation history and real production data, while stabilizing judges through ensemble evaluations, repeated runs to calculate standard deviation, and binary scoring axes.", "body_md": "*Traditional tests are binary. LLM evaluations are a forecast.*\n\nWe spent three weeks chasing ghosts in our prompt templates. We’d run our benchmark, get an 84%, tweak a word, and see it drop to 81%. We’d revert the change, run it again, and watch it swing to 87%.\n\nNothing in the code had changed.\n\nLLMs aren’t unit tests. They’re probabilistic, but we treated them like compiled binaries. When your test suite is a forecast, a single scoreboard is a lie.\n\n## Why scoreboards lie\n\nA score looks like a measurement. It isn’t. It’s an estimate.\n\nIf you’ve run A/B tests or performance benchmarks, you’ve dealt with statistical noise. LLMs just make it impossible to ignore.\n\nTo build an eval system you can trust, you must separate the noise. It comes from two places:\n\nWhen the inputs are ambiguous, fix the inputs. When the judge is inconsistent, fix the judge. The hardest part is knowing which problem you have.\n\nEvery score change asks: Did the model change? Did the prompt change? Or did the measurement change? Treating them all as “the score changed” is how evaluation pipelines lose credibility.\n\n## Problem 1: The input is ambiguous\n\nImagine a user replies with a single word: “Stop”.\n\nFor an SMS bot, that’s an opt-out. For customer support, they might be frustrated or just pausing the session. Without context, both are correct. Even human evaluators will disagree. The ambiguity is baked into the conversation itself.\n\nStatisticians call this aleatoric uncertainty. In plain English: the input lacks the information needed for a single correct answer. Improving the judge won’t fix this. You’re optimizing the wrong side of the equation.\n\n### Fixing the inputs\n\nTo clean up inputs, we changed how we built benchmarks:\n\n- We passed the last three turns of history. Single-turn conversations often lack the context needed for a reliable evaluation.\n- We stopped writing synthetic test cases. Now, we strip personal data from real production sessions and use those as fixtures.\n- We tagged prompts with multiple valid replies, preventing the judge from penalizing correct but unexpected answers.\n\n## Problem 2: The judge is inconsistent\n\nNow look at the other side: a clean prompt and a correct model output, but the LLM judge gives it a 3 out of 5. Five minutes later, the exact same output gets a 5.\n\nNothing changed, but the score did. This is epistemic uncertainty: noise in the measurement, not the task.\n\nThis noise usually comes from three places:\n\n- Hosted model nondeterminism. Even at temperature 0, hosted models can return different outputs over time due to backend updates or load balancing.\n- Vague rubrics. If you ask a judge whether a response is “helpful,” it must invent a definition on every run.\n- Model bias. LLMs favor their own writing, prefer longer answers, and lean toward the first option in a list.\n\n### Fixing the judge\n\nHere is how we stabilized our evaluations:\n\n- We ensemble evaluations across multiple models (Claude, GPT-4, Gemini) and flag the run if they disagree.\n- We run the same suite three times in a row to calculate baseline standard deviation. An 84% score is never a static number; it’s\n**84% ± 3%**. - We score along binary, objective axes. Instead of asking for a 1-to-5 rating, we ask yes-or-no questions: Did it answer? Did it hallucinate? Was the tone polite?\n\n## Designing a better eval architecture\n\nIf you don’t know where your noise is coming from, you’ll waste weeks tuning the wrong thing. You can’t prompt-engineer your way out of an ambiguous input, and you can’t fix hosted model nondeterminism by rewriting your benchmark.\n\nOnce we separated the noise, our design fell into place. We stopped trying to hit 100% correctness and focused on confidence intervals. We ensembled evaluations to flag judge drift, and we stopped treating variance as a bug and started treating it as a property of the system.\n\n## Stop chasing the scoreboard\n\nUnit tests are binary. LLM evaluations are more like performance benchmarks or A/B tests: they don’t prove correctness, they estimate confidence.\n\nOnce you stop treating evals like unit tests, you stop chasing minor fluctuations. The goal isn’t a perfect green checkmark. It’s to build enough statistical confidence to ship.", "url": "https://wpnews.pro/news/the-two-sources-of-noise-every-llm-evaluation-system-must-handle", "canonical_source": "https://wilburhimself.github.io/blog/64-the-two-sources-of-noise-every-llm-evaluation-system-must-handle/", "published_at": "2026-07-26 05:57:24+00:00", "updated_at": "2026-07-26 06:23:11.854505+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research"], "entities": ["Claude", "GPT-4", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/the-two-sources-of-noise-every-llm-evaluation-system-must-handle", "markdown": "https://wpnews.pro/news/the-two-sources-of-noise-every-llm-evaluation-system-must-handle.md", "text": "https://wpnews.pro/news/the-two-sources-of-noise-every-llm-evaluation-system-must-handle.txt", "jsonld": "https://wpnews.pro/news/the-two-sources-of-noise-every-llm-evaluation-system-must-handle.jsonld"}}