{"slug": "how-to-correct-an-llm-judge-s-pass-rate-with-100-human-labels", "title": "How to Correct an LLM Judge's Pass Rate With 100 Human Labels", "summary": "An engineer described a method for correcting an LLM judge's reported pass rate using roughly 100 human-labeled outputs, applying the Rogan-Gladen correction borrowed from medical testing. The approach measures the judge's true positive rate (share of good outputs it passes) and true negative rate (share of bad outputs it fails), then inverts the blended score to recover the product's real pass rate. The engineer warned that lenient judges, such as one 2025 study finding judges passed 96% of good answers but failed fewer than a quarter of bad ones, both inflate scores and mask real progress, and that dividing by a small TPR+TNR-1 term amplifies label noise.", "body_md": "**Engineer's view.** This is a health check that returns 200 while the database is down. Your uptime number then mixes real uptime with the check's blind spot. A model grading your outputs does the same thing. It passes some bad ones, so the score runs high and every real fix looks smaller than it was.\n\n**TL;DR.** The pass rate a grading model reports is part your product and part the grader's own mistakes. Measure those mistakes on a small set a person has checked, and you can subtract them back out.\n\nYou run a fruit stand, and a helper sorts apples into a good basket and a bad basket. She\n\nis quick, but she is kind. Most bruised apples land in the good basket too.\n\nSo you count the good basket and feel proud. Nine apples in ten look fine. That number is\n\npartly your apples and partly her kindness.\n\nTo find the real number, you test her. Hand her fifty apples you know are good and fifty\n\nyou know are bruised. Count how many of each she gets right. Now you know how many bruised\n\napples she lets through on a normal day. Take that many out of the good basket, and what is\n\nleft is the true count.\n\nYou have shipped this bug already, in a system with no model in it.\n\nYour uptime dashboard said 99.9% for a month. Users kept filing tickets about errors. After\n\na day of digging you found the health check. It pinged an endpoint that returned 200 as long\n\nas the web process was alive. When the database was down, the check still passed.\n\nNothing was wrong with the arithmetic. The dashboard faithfully counted what the check said.\n\nBut the check had a blind spot, so the number mixed two things: real uptime and the check's\n\nhabit of passing broken states. You could not fix the dashboard until you knew how often the\n\ncheck was wrong.\n\nAn eval that uses an **LLM judge** has that bug at full size. An LLM judge is a model you\n\nprompt to grade each output of your feature as Pass or Fail. Its pass rate goes on the\n\ndashboard as if it were your product's pass rate. It is not. Judges make mistakes in both\n\ndirections, and they lean lenient. One 2025 study of judges grading feedback on student code\n\nfound they passed 96% of good answers but failed fewer than a quarter of bad ones.\n\nThat leniency does two kinds of damage. The score runs high. Worse, it hides progress. When\n\nthe judge passes most bad outputs anyway, fixing some of them barely moves the number.\n\n**The fix: label about a hundred outputs by hand, measure how often the judge agrees on the good ones and on the bad ones, and use those two rates to undo the blend.**\n\nFrom two numbers the dashboard never shows. The **TPR** is the share of truly good outputs\n\nthe judge passes. The **TNR** is the share of truly bad outputs the judge fails. Here, Pass\n\ncounts as the positive answer.\n\nThe judge's pass rate is the good outputs it passes plus the bad outputs it lets through. A\n\nstrict judge with a TPR of 0.92 and a TNR of 0.88 turns a true 70% into 68%. That is close,\n\nbut only by luck: its two mistakes happen to cancel.\n\nThe **Rogan-Gladen correction** runs that arithmetic backwards. It comes from medical\n\ntesting, where a lab test has the same two error rates. Add the TNR to the judge's pass\n\nrate, subtract one, and divide by the TPR plus the TNR minus one.\n\nBecause only part of any real change gets through. That part is the TPR plus the TNR minus\n\none. For the strict judge it is 0.80, so a real eight-point gain shows as 6.4 points.\n\nA lenient judge with a TNR of 0.24 keeps only a fifth of the change. Your team ships a real\n\nimprovement, and the chart moves by 1.6 points.\n\nNo, and this is the part that bites. The correction divides by that same 0.20. Dividing by a\n\nsmall number turns small mistakes in your labels into huge swings. With a hundred labels, the\n\nlenient judge's corrected rate could be anything from 32% to 100%. Fix the judge first.\n\n**When this matters.** Any time a model grades your outputs and you report the share it\n\npassed. That covers a nightly eval, a release check, and a quality chart on a dashboard. It\n\nmatters most when you compare two numbers: this release against the last one, or one prompt\n\nagainst another.\n\n**How it affects you.** The level is off, usually too high. The comparisons are shrunk, so\n\nreal improvements look small and a prompt change that helped can look like noise. And it\n\ngets worse quietly. Swap in a stricter judge model, and your pass rate drops on a day you\n\nshipped nothing.\n\n**What to do about it.** Start with ten minutes and no new tooling. Find ten outputs you\n\nknow are bad. A thumbs-down from a user, a bug report, or a few you break on purpose all\n\nwork. Run your judge on them and count how many it passes. If it passes four, your judge\n\nlets roughly 40% of failures through. Every bad output it misses is counted on your dashboard\n\nas a success.\n\nThen do it properly. Label about a hundred outputs drawn from real traffic, half good and half\n\nbad. Measure the judge's TPR and TNR on them. Report the corrected rate with an interval\n\nnext to the raw one.\n\nIf the TPR plus the TNR comes out under about 0.3, stop. Do not correct. Rewrite the judge\n\nprompt or narrow it to one failure mode, and measure again. The next section shows the three\n\npieces of code, and the check that tells you they work.\n\n**The change.**\n\nThree people touch this. Each writes a little code, and none of it is hard. Getting the\n\nconvention wrong is the usual failure, so name it once and use it everywhere.\n\n*Role 1, whoever owns the labels.* Keep the human verdicts beside the judge verdicts, one\n\nline per output, sampled from production. Balance the classes on purpose. Real traffic\n\nmight be 90% good, and ten bad examples cannot measure a TNR.\n\n```\n{\"id\": \"conv-8812\", \"human\": 1, \"judge\": 1, \"split\": \"dev\"}\n{\"id\": \"conv-9034\", \"human\": 0, \"judge\": 1, \"split\": \"test\"}\n{\"id\": \"conv-9107\", \"human\": 0, \"judge\": 0, \"split\": \"test\"}\n```\n\nUse the `dev` rows while you rewrite the judge prompt. Measure the final rates on the `test`\n\nrows only, once. Rates measured on the rows you tuned against will look better than they are.\n\n*Role 2, whoever owns the eval script.* Compute the two rates, correct the pass rate, and\n\nrefuse when the judge is too weak.\n\n``` python\ndef judge_rates(human, judge):          # 1 = Pass, for both lists\n    good = [j for h, j in zip(human, judge) if h == 1]\n    bad = [j for h, j in zip(human, judge) if h == 0]\n    return sum(good) / len(good), 1 - sum(bad) / len(bad)\n\ndef corrected_pass_rate(p_obs, tpr, tnr, min_youden=0.30):\n    youden = tpr + tnr - 1\n    if youden < min_youden:\n        return None                     # report \"judge too weak\", not a number\n    return min(1.0, max(0.0, (p_obs + tnr - 1) / youden))\n```\n\nThe interval comes from resampling. Redraw the labeled rows and the production verdicts with\n\nreplacement, recompute both rates and the corrected rate, and repeat 2,000 times. The middle\n\n95% of the results is the interval. To compare two releases, draw one pair of rates per round\n\nand use it for both. The judge's error is shared, so part of it cancels. The full version is in\n\n[`code_example.py`](https://github.com/magna56/aI_daily_run/blob/main/2026-10-26/code_example.py). If you would rather not write it, the MIT-licensed `judgy` package does the\n\nsame thing:\n\n``` python\nfrom judgy import estimate_success_rate\ntheta, low, high = estimate_success_rate(\n    test_labels=human, test_preds=judge, unlabeled_preds=production_verdicts)\n```\n\n*Role 3, whoever owns the dashboard and the review.* Show the raw rate, the corrected rate,\n\nand its interval. Store the judge's TPR, TNR and exact model version next to each run. A\n\nnew judge prompt or model needs new rates before its numbers can be compared.\n\n**How you know it worked.**\n\nRun two checks. First, use the `dev` rows to measure the rates, then correct the judge's pass\n\nrate on the `test` rows. The corrected number should land near the human pass rate on those\n\nrows. The raw number usually will not.\n\nSecond, swap the convention on purpose. In the Code tab, a corrected 70.0% becomes 75.0% when\n\nTPR and TNR trade places. Some guides call a caught failure the positive case, so their TPR\n\nis your TNR. If two people's numbers disagree by a few points, check this first.\n\nWhen the judge is barely better than a coin. The correction divides by the judge's two rates\n\nadded together, minus one. Near zero, the interval covers almost everything. Improve the judge\n\nuntil both rates clear 0.8, then correct.\n\nWhen today's failures are a new kind. The correction assumes the judge misses failures as often\n\nin production as it did on your labels. In the Code tab, the labels say the judge fails 88% of\n\nbad outputs. Today's failures are subtler, and it fails only 60% of them. The true rate is 60%,\n\nand the stale correction reports 74%. Re-label a fresh sample when the product changes.\n\nWhen you need a verdict on one output. The correction fixes a rate across many outputs. It says\n\nnothing about whether a given reply was good, so it cannot gate a single response.\n\nAnd when you can afford to label everything. At fifty outputs a week, a person is the eval.\n\nThree questions before you adopt it:\n\n*Originally published at [theaicommit.com](https://theaicommit.com/2026-10-26-how-to-correct-an-llm-judges-pass-rate-with-100-human-labels/), a daily AI engineering lab for software engineers. The original has an [interactive version](https://theaicommit.com/2026-10-26-how-to-correct-an-llm-judges-pass-rate-with-100-human-labels/) you can play with, and the full runnable code is on [GitHub](https://github.com/magna56/aI_daily_run/blob/main/2026-10-26/code_example.py).*", "url": "https://wpnews.pro/news/how-to-correct-an-llm-judge-s-pass-rate-with-100-human-labels", "canonical_source": "https://dev.to/theaicommit/how-to-correct-an-llm-judges-pass-rate-with-100-human-labels-16oa", "published_at": "2026-10-26 12:05:05+00:00", "updated_at": "2026-10-26 12:13:27.469294+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "artificial-intelligence"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-correct-an-llm-judge-s-pass-rate-with-100-human-labels", "markdown": "https://wpnews.pro/news/how-to-correct-an-llm-judge-s-pass-rate-with-100-human-labels.md", "text": "https://wpnews.pro/news/how-to-correct-an-llm-judge-s-pass-rate-with-100-human-labels.txt", "jsonld": "https://wpnews.pro/news/how-to-correct-an-llm-judge-s-pass-rate-with-100-human-labels.jsonld"}}