Engineer's view. This is a health check that returns 200 while the database is down. Your uptime number then mixes real uptime with the check's blind spot. A model grading your outputs does the same thing. It passes some bad ones, so the score runs high and every real fix looks smaller than it was.
TL;DR. The pass rate a grading model reports is part your product and part the grader's own mistakes. Measure those mistakes on a small set a person has checked, and you can subtract them back out.
You run a fruit stand, and a helper sorts apples into a good basket and a bad basket. She
is quick, but she is kind. Most bruised apples land in the good basket too.
So you count the good basket and feel proud. Nine apples in ten look fine. That number is
partly your apples and partly her kindness.
To find the real number, you test her. Hand her fifty apples you know are good and fifty
you know are bruised. Count how many of each she gets right. Now you know how many bruised
apples she lets through on a normal day. Take that many out of the good basket, and what is
left is the true count.
You have shipped this bug already, in a system with no model in it.
Your uptime dashboard said 99.9% for a month. Users kept filing tickets about errors. After
a day of digging you found the health check. It pinged an endpoint that returned 200 as long
as the web process was alive. When the database was down, the check still passed.
Nothing was wrong with the arithmetic. The dashboard faithfully counted what the check said.
But the check had a blind spot, so the number mixed two things: real uptime and the check's
habit of passing broken states. You could not fix the dashboard until you knew how often the
check was wrong.
An eval that uses an LLM judge has that bug at full size. An LLM judge is a model you
prompt to grade each output of your feature as Pass or Fail. Its pass rate goes on the
dashboard as if it were your product's pass rate. It is not. Judges make mistakes in both
directions, and they lean lenient. One 2025 study of judges grading feedback on student code
found they passed 96% of good answers but failed fewer than a quarter of bad ones.
That leniency does two kinds of damage. The score runs high. Worse, it hides progress. When
the judge passes most bad outputs anyway, fixing some of them barely moves the number.
The fix: label about a hundred outputs by hand, measure how often the judge agrees on the good ones and on the bad ones, and use those two rates to undo the blend.
From two numbers the dashboard never shows. The TPR is the share of truly good outputs
the judge passes. The TNR is the share of truly bad outputs the judge fails. Here, Pass
counts as the positive answer.
The judge's pass rate is the good outputs it passes plus the bad outputs it lets through. A
strict judge with a TPR of 0.92 and a TNR of 0.88 turns a true 70% into 68%. That is close,
but only by luck: its two mistakes happen to cancel.
The Rogan-Gladen correction runs that arithmetic backwards. It comes from medical
testing, where a lab test has the same two error rates. Add the TNR to the judge's pass
rate, subtract one, and divide by the TPR plus the TNR minus one.
Because only part of any real change gets through. That part is the TPR plus the TNR minus
one. For the strict judge it is 0.80, so a real eight-point gain shows as 6.4 points.
A lenient judge with a TNR of 0.24 keeps only a fifth of the change. Your team ships a real
improvement, and the chart moves by 1.6 points.
No, and this is the part that bites. The correction divides by that same 0.20. Dividing by a
small number turns small mistakes in your labels into huge swings. With a hundred labels, the
lenient judge's corrected rate could be anything from 32% to 100%. Fix the judge first.
When this matters. Any time a model grades your outputs and you report the share it
passed. That covers a nightly eval, a release check, and a quality chart on a dashboard. It
matters most when you compare two numbers: this release against the last one, or one prompt
against another.
How it affects you. The level is off, usually too high. The comparisons are shrunk, so
real improvements look small and a prompt change that helped can look like noise. And it
gets worse quietly. Swap in a stricter judge model, and your pass rate drops on a day you
shipped nothing.
What to do about it. Start with ten minutes and no new tooling. Find ten outputs you
know are bad. A thumbs-down from a user, a bug report, or a few you break on purpose all
work. Run your judge on them and count how many it passes. If it passes four, your judge
lets roughly 40% of failures through. Every bad output it misses is counted on your dashboard
as a success.
Then do it properly. Label about a hundred outputs drawn from real traffic, half good and half
bad. Measure the judge's TPR and TNR on them. Report the corrected rate with an interval
next to the raw one.
If the TPR plus the TNR comes out under about 0.3, stop. Do not correct. Rewrite the judge
prompt or narrow it to one failure mode, and measure again. The next section shows the three
pieces of code, and the check that tells you they work.
The change.
Three people touch this. Each writes a little code, and none of it is hard. Getting the
convention wrong is the usual failure, so name it once and use it everywhere.
Role 1, whoever owns the labels. Keep the human verdicts beside the judge verdicts, one
line per output, sampled from production. Balance the classes on purpose. Real traffic
might be 90% good, and ten bad examples cannot measure a TNR.
{"id": "conv-8812", "human": 1, "judge": 1, "split": "dev"}
{"id": "conv-9034", "human": 0, "judge": 1, "split": "test"}
{"id": "conv-9107", "human": 0, "judge": 0, "split": "test"}
Use the dev rows while you rewrite the judge prompt. Measure the final rates on the test
rows only, once. Rates measured on the rows you tuned against will look better than they are.
Role 2, whoever owns the eval script. Compute the two rates, correct the pass rate, and
refuse when the judge is too weak.
def judge_rates(human, judge): # 1 = Pass, for both lists
good = [j for h, j in zip(human, judge) if h == 1]
bad = [j for h, j in zip(human, judge) if h == 0]
return sum(good) / len(good), 1 - sum(bad) / len(bad)
def corrected_pass_rate(p_obs, tpr, tnr, min_youden=0.30):
youden = tpr + tnr - 1
if youden < min_youden:
return None # report "judge too weak", not a number
return min(1.0, max(0.0, (p_obs + tnr - 1) / youden))
The interval comes from resampling. Redraw the labeled rows and the production verdicts with
replacement, recompute both rates and the corrected rate, and repeat 2,000 times. The middle
95% of the results is the interval. To compare two releases, draw one pair of rates per round
and use it for both. The judge's error is shared, so part of it cancels. The full version is in
code_example.py. If you would rather not write it, the MIT-licensed judgy package does the
same thing:
from judgy import estimate_success_rate
theta, low, high = estimate_success_rate(
test_labels=human, test_preds=judge, unlabeled_preds=production_verdicts)
Role 3, whoever owns the dashboard and the review. Show the raw rate, the corrected rate,
and its interval. Store the judge's TPR, TNR and exact model version next to each run. A
new judge prompt or model needs new rates before its numbers can be compared.
How you know it worked.
Run two checks. First, use the dev rows to measure the rates, then correct the judge's pass
rate on the test rows. The corrected number should land near the human pass rate on those
rows. The raw number usually will not.
Second, swap the convention on purpose. In the Code tab, a corrected 70.0% becomes 75.0% when
TPR and TNR trade places. Some guides call a caught failure the positive case, so their TPR
is your TNR. If two people's numbers disagree by a few points, check this first.
When the judge is barely better than a coin. The correction divides by the judge's two rates
added together, minus one. Near zero, the interval covers almost everything. Improve the judge
until both rates clear 0.8, then correct.
When today's failures are a new kind. The correction assumes the judge misses failures as often
in production as it did on your labels. In the Code tab, the labels say the judge fails 88% of
bad outputs. Today's failures are subtler, and it fails only 60% of them. The true rate is 60%,
and the stale correction reports 74%. Re-label a fresh sample when the product changes.
When you need a verdict on one output. The correction fixes a rate across many outputs. It says
nothing about whether a given reply was good, so it cannot gate a single response.
And when you can afford to label everything. At fifty outputs a week, a person is the eval.
Three questions before you adopt it:
Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.