{"slug": "how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you", "title": "How to calibrate LLM as a judge before your eval dashboard lies to you", "summary": "Teams running LLM-as-a-judge evaluations should hand-label 50 to 100 real outputs, including at least 20 failures, and measure the judge's agreement with those labels before trusting any eval dashboard score, according to a calibration guide published by Startup Fortune. The guide cites the 2023 LMSYS paper 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,' which found GPT-4 agreed with human preferences more than 80% of the time but also documented position bias (GPT-4 gave consistent verdicts about 65% of the time when answer order was swapped, Claude-v1 about a quarter of the time), verbosity bias (Claude-v1 and GPT-3.5 were fooled by padded answers in more than 90% of cases, GPT-4 under 9%), and self-enhancement (roughly a 10-point win-rate bump for GPT-4 and about 25 for Claude-v1). The guide recommends printing Cohen's kappa next to every eval score and rerunning the labelled set whenever the judge model or prompt changes, noting that 86 matches out of 100 labels can still be a kappa of only 0.46.", "body_md": "*Your eval dashboard says 91% and nobody has checked whether the grader agrees with a person. A one-afternoon test shows how much of that number you can believe.*\n\n- Hand-label 50 to 100 real outputs, with at least 20 failures among them, and treat that set as the exam your LLM judge has to pass.\n- Raw agreement flatters a judge: 86 matches out of 100 labels can still be a Cohen's kappa of only 0.46.\n- Swap answer order, pad answers with filler, and grade with a different model family than the one that wrote the answers to catch position, verbosity and self-preference bias.\n- The 2023 MT-Bench study found GPT-4 agreed with humans over 80% of the time, but that result says nothing about your rubric or your data.\n- Print the kappa next to every eval score, and rerun the labelled set whenever the judge model or prompt changes.\n\nThe method for how to calibrate LLM as a judge fits in one sentence: label 50 to 100 outputs by hand, run your judge on the same outputs, and measure how well the two sets of verdicts agree. Almost nobody does it. The usual path is to write a grading prompt, point a model at it, and watch a dashboard fill up with scores to one decimal place. Nothing in that process tells you whether 0.83 means anything.\n\nFrankly, an uncalibrated judge is a random number generator with good manners. It produces tidy scores, they move up and down when you change your product, and you make roadmap decisions off them. If the judge disagrees with your users half the time, you're steering by noise.\n\nThe research everyone leans on is the 2023 paper from the LMSYS team, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. It reported that GPT-4 agreed with human preferences more than 80% of the time, roughly the same rate at which two humans agree with each other. That's a real result, and it's why judges became standard. But it was measured on general chat answers, with a strong judge, on a benchmark built for that purpose. Your judge grades support replies, contract summaries or SQL against a rubric the paper never saw. You have to measure it yourself.\n\nThe same paper also documented the ways judges go wrong, and those are worth knowing before you build anything.\n\n[How to Set LLM Eval Pass Threshold Before the Demo Does It for You](https://startupfortune.com/how-to-set-llm-eval-pass-threshold-before-the-demo-does-it-for-you/)\n\nHow to set an LLM eval pass threshold comes down to three inputs: what a wrong answer costs, how often trained humans disagree on the same task, and how wide your confidence interval is. Air Canada and Klarna show what happens when teams skip one. - [how to set LLM eval pass thresholds](https://startupfortune.com/how-to-set-llm-eval-pass-threshold-before-the-demo-does-it-for-you/) - [choosing LLM evaluation thresholds before demos](https://startupfortune.com/how-to-set-llm-eval-pass-threshold-before-the-demo-does-it-for-you/)\n\nPosition bias came first. When the researchers swapped the order of two answers, GPT-4 gave a consistent verdict about 65% of the time, and Claude-v1 only about a quarter of the time. The judge was often reacting to which answer came first, not which one was better.\n\nVerbosity was next. The authors ran what they called a repetitive list attack: they padded an answer with rephrased versions of the same points and checked whether the judge rewarded it. Claude-v1 and GPT-3.5 were fooled in more than 90% of cases, while GPT-4 was fooled in under 9%. They also saw self-enhancement, where models favoured answers written by themselves, with a win-rate bump of roughly 10 points for GPT-4 and about 25 for Claude-v1.\n\nThose were 2023 models, and current ones are better behaved. Don't take that on faith either. The point of the paper's design is that every one of these biases can be tested with a cheap experiment, and you can run the same experiments on your own judge in an hour.\n\n## Build the LLM judge golden dataset first\n\nStart with real traces, not synthetic ones. Pull outputs your product actually produced, from production logs or a staging run with real inputs. Synthetic examples are cleaner than reality, and a judge that handles them says little about what it'll do with your customers' messy prompts.\n\nAim for 100 labelled examples. Fifty will do if that's all your time allows, but under that the numbers get too jumpy to read. The more important rule is the mix. If your product fails 5% of the time and you sample at random, 100 examples gets you about five failures, and you can't measure a judge's ability to catch failures from five cases. Go looking for bad outputs on purpose, through user complaints, thumbs-down clicks, and retries, until you have at least 20 failures and 20 passes.\n\nLabel with a binary pass or fail, and write down what each one means before you start. Hamel Husain, the machine learning engineer who runs a widely followed course on evals, has argued for binary judgments over 1 to 5 scales for exactly this reason: nobody can say what the difference between a 3 and a 4 is, and your labellers and your judge will each pick a different answer. A pass/fail definition forces the argument about quality to happen once, in writing, instead of silently inside every grade.\n\nHave one person with real domain knowledge do the labelling. A founder who knows what a good answer looks like in their own product is often the right person. Write one sentence of reasoning beside each label. Those sentences become your few-shot examples later and, more usefully, they expose where your own rubric is vague.\n\n[How to Build an Eval Set for AI Agents Before You Trust One in Production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/)\n\nAir Canada was ordered to pay about C$812 in February 2024 after its chatbot invented a refund policy. This guide covers where to find test cases, how to score them with code before an LLM judge, and why each case should run three to five times. - [build an eval set for AI agents](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) - [AI agent test cases before production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/)\n\nIf you can spare the time, have a second person label 30 of the same examples without seeing the first person's answers. Their agreement with each other is your ceiling. If two humans only reach a kappa of 0.6, a judge that reaches 0.6 is already doing as well as a person, and demanding more from the machine is unfair.\n\nThen split the set. Keep 20 to 30 examples as a development pile you can use to tune the judge's prompt, and lock the remainder away as a test pile. If you rewrite your prompt while looking at every labelled example, you'll end up with a judge that's memorised your labels, and the score you report will be fiction again.\n\n## Measure LLM judge human agreement with Cohen's kappa\n\nRun the judge on the test pile and compare. The first number everyone reaches for is plain agreement: the share of examples where judge and human gave the same verdict. It's the wrong number to stop at, because a judge that says pass to nearly everything will look good whenever most outputs genuinely pass.\n\nHere's a worked example with made-up but realistic numbers. You label 100 outputs. Humans pass 80 and fail 20. Your judge passes 90 and fails 10. Both say pass on 78 of them, both say fail on 8, the judge passes 12 that humans failed, and the judge fails 2 that humans passed. Agreement is 86%, which sounds respectable.\n\nCohen's kappa corrects for the agreement you'd get by luck. Take observed agreement (0.86), subtract the chance agreement, and divide by what's left over after chance. Chance agreement here is 0.80 times 0.90 plus 0.20 times 0.10, which is 0.74. So kappa is (0.86 minus 0.74) divided by (1 minus 0.74), or about 0.46. The same judge that looked 86% accurate lands in the moderate band of the Landis and Koch scale that most people use to read kappa: 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, above 0.80 almost perfect.\n\nThe kappa is one thing to report, but the split inside it matters more. Compute the true positive rate, meaning how often the judge passes what humans pass, and the true negative rate, how often it fails what humans fail. In the example the judge passes 78 of the 80 good outputs, a rate of 0.975. It fails only 8 of the 20 bad ones, a rate of 0.40. That's a judge that's easy on everything, and you'd never see it from the headline score.\n\nMy thresholds, and you should set your own: below 0.4, don't use the judge for decisions at all. Between 0.4 and 0.6, use it to rank experiments and look at trends, but don't trust its absolute numbers. At 0.7 or above, it's good enough to gate a release or alert on a regression. A kappa of 0.46 doesn't mean throw the judge away. It means the dashboard is a rough guide, and you now know that.\n\nThere's a second use for the same numbers. Using the true positive and true negative rates, you can correct a judge's headline score with the Rogan-Gladen estimate: take the judge's observed pass rate, add the true negative rate, subtract one, and divide by the true positive rate plus the true negative rate minus one. In the example that's (0.90 plus 0.40 minus 1) divided by (0.975 plus 0.40 minus 1), which gives 0.80, exactly the human pass rate. The dashboard said 90%. The honest number was 80%. The correction only works when the two rates add up to well over one, so a judge near coin-flip quality can't be rescued by arithmetic.\n\nDon't skip the error bars either. With 100 labels, the 95% interval on an 86% agreement rate is about plus or minus 7 points, and the interval on kappa is wider than that. A judge that scores 0.52 on one batch and 0.44 on another hasn't necessarily changed. It's the sample talking. This is the main reason to go toward 100 labels rather than stopping at 50.\n\n## Test LLM as judge evaluation bias on purpose\n\nAgreement with humans on your test pile tells you the judge is right on average. Bias tests tell you whether it's right for the wrong reasons, and each takes a few minutes of code.\n\nFor position bias, if you use pairwise comparisons, run every pair twice with the order swapped and count how often the verdict flips. A flip means the judge was reacting to position. Take the flip rate seriously: if 30% of verdicts reverse when you swap A and B, then 30% of your pairwise results are coin tosses. The simple fix is to run both orders and only count a win when the judge agrees with itself, scoring the rest as ties.\n\nFor verbosity bias, take 20 outputs that passed and make them longer without making them better: add a restating opening paragraph, a closing summary, a couple of hedging sentences. Rerun the judge. If scores go up on answers that contain no new information, the judge likes length. You can also check the correlation between output length and judge score across your whole dataset, then check it against human labels. If the judge's scores track word count more tightly than the humans' do, you've found your culprit.\n\nFor self-preference, check which model wrote the outputs. If GPT-4-class models generate your answers and a model from the same family grades them, expect some flattering. The MT-Bench numbers above suggest the effect is real, and the cheap defence is to grade with a model from a different family, then compare. If you have the labels, you can see it directly: split the test pile by generator and see whether the judge's agreement with humans is lower on outputs from other vendors than on its own.\n\nOne more check costs nothing. Ask the judge for its reasoning before the verdict, then read 20 of those explanations next to the human reasoning notes you wrote earlier. Disagreements usually cluster around a single misread rule in your rubric, and fixing that sentence in the prompt often lifts kappa more than swapping to a bigger model.\n\n## How to validate an LLM evaluator when anything changes\n\nThe calibration set isn't a one-off. It's a regression test for the grader itself. Every time you change the judge's prompt, swap the underlying model, or ship a new version of the product that produces different kinds of outputs, rerun the test pile and compare. A provider quietly updating a model can move your judge's behaviour without any change in your code, and the only way to catch it is to keep measuring against the same labelled examples.\n\nAdd to the set as you go. Once a month, pull 20 fresh production traces, label them yourself, and fold them in. The most useful additions are cases where the judge and a user disagreed, because those are the ones your labelled set was missing. Over a year, you'd have a few hundred examples and a much tighter confidence interval.\n\nThe effort is small and the cost is mostly your time. A person labelling at one to two minutes per example finishes 100 in two to three hours, and running a judge over 100 outputs costs pennies. Against that, consider what an uncalibrated judge can do: it can tell you a prompt change improved quality by four points when it didn't, and you'll ship it, tell your investors, and build on it.\n\nThe habit I'd adopt is to print the agreement number beside every score the dashboard shows. Something like: helpfulness 0.82, judge agreement kappa 0.46 on 100 labels. It takes ten seconds to add and it changes the conversation in a team meeting, because nobody argues over a two-point movement on a metric whose own reliability is printed right next to it.\n\nIf your first kappa comes back at 0.46, as in the example, you haven't wasted anything. You've learned that your dashboard was a rough guide, and you're holding 100 labelled examples to improve it with.\n\n**Also read:** [Build an LLM model deprecation migration plan before the cutoff week](https://startupfortune.com/build-an-llm-model-deprecation-migration-plan-before-the-cutoff-week/) • [How to Set LLM Eval Pass Threshold Before the Demo Does It for You](https://startupfortune.com/how-to-set-llm-eval-pass-threshold-before-the-demo-does-it-for-you/) • [Europe, EU, European Union and Schengen Explained in 60 Seconds](https://startupfortune.com/europe-eu-european-union-and-schengen-explained-in-60-seconds/)\n\n*This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.*\n\n## Join the discussion\n\n[Open in the community →](https://startupfortune.com/community/)\n\nAlmost there. Sign in and your reply posts straight away.", "url": "https://wpnews.pro/news/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you", "canonical_source": "https://startupfortune.com/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you/", "published_at": "2026-11-21 07:07:13+00:00", "updated_at": "2026-11-21 07:12:18.503449+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-agents"], "entities": ["Startup Fortune", "LMSYS", "GPT-4", "Claude-v1", "GPT-3.5", "MT-Bench", "Chatbot Arena"], "also_reported_by": [], "takeaway": "Hand-label 50 to 100 real outputs with at least 20 failures and report Cohen's kappa alongside raw agreement, since 86 of 100 matches can still be a kappa of only 0.46.", "recommendation": {"advice": "Hand-label 50 to 100 real outputs with at least 20 failures, then rerun the labelled set and print Cohen's kappa next to every eval score whenever the judge model or prompt changes.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you", "markdown": "https://wpnews.pro/news/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you.md", "text": "https://wpnews.pro/news/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you.txt", "jsonld": "https://wpnews.pro/news/how-to-calibrate-llm-as-a-judge-before-your-eval-dashboard-lies-to-you.jsonld"}}