If AlpacaEval’s GPT-4 Turbo judge used one sampled verdict per answer, GPT-4o’s error bar would be ±3.45 points. Five verdicts make it ±3.00. So do 32% more questions.
Grading every answer 5 times with your LLM judge costs 5x the judge bill. On 805 public AlpacaEval questions, the judge’s own odds show it would shrink the error bar by only 13%.
The idea comes from Netflix staff research scientist Cameron Wolfe. After this article you can run one script on your judge’s grades and see whether to pay for extra judge runs or more test cases.
Cameron Wolfe, who also writes the Deep (Learning) Focus newsletter, gave the talk “How Netflix Evaluates AI When There’s No Right Answer”, which the AI Engineer channel posted on November 5. It covers tasks like creative writing, where pass or fail is a judgment call.
His framework has three parts:
His main thread is noise: both the model and its LLM judge are random. His fixes are power analysis to size the eval, error bars on every number, and resampling, meaning multiple rollouts from the model or the judge. Below, I work out what judge resampling buys from a judge whose odds are public.
You need a judge that graded many questions and said how sure it was. AlpacaEval 2.0 publishes that. Its GPT-4 Turbo judge compares each model’s answer with GPT-4 Turbo’s own answer on 805 questions, and the repo stores the judge’s win probability for every question, read from its token odds.
That probability is what makes the test possible. The judge’s config asks for one token at temperature 1, so if it says 0.7 on a question, one sampled verdict is literally a coin that lands “win” 70% of the time. Its spread, 0.7 × 0.3, is the judge’s own noise on that question. The spread of the probabilities across questions is the noise from which questions you happened to pick.
Your eval score is an average over questions, so its error bar has two sources:
+--------+--------------------+-----------------+| source | what it is | cure |+========+====================+=================+| tasks | which questions | more questions || judge | a coin per verdict | more judge runs |+--------+--------------------+-----------------+
Grading each answer K times divides only the judge part by K. The task part does not move.
judgenoise.py is standard-library Python 3.8+. It reads either an AlpacaEval annotations.json or a CSV of your own grades, splits the variance into those two parts, and prints the error bar for each number of judge runs:
#!/usr/bin/env python3"""judgenoise.py - how much of your eval's error bar is the judge?Usage: judgenoise.py FILE [K ...]FILE is an AlpacaEval annotations.json (preference 1..2) or a CSV oftask,grade lines (grade 0..1; repeat a task once per judge run).K values: judge runs per answer to price (default 1 2 3 5 10).Stdlib only, Python 3.8+."""import csv, json, math, sysdef load(path): # returns [(task mean, judge variance, runs behind the mean)], basis if path.endswith(".json"): rows = json.load(open(path)) # AlpacaEval: preference 1 = baseline wins, 2 = model wins ps = [float(r["preference"]) - 1 for r in rows if r.get("preference") not in (None, "")] # the judge's own odds: one sampled verdict has variance p(1-p) return [(p, p * (1 - p), math.inf) for p in ps], "judge token odds" by = {} with open(path, newline="") as f: for i, row in enumerate(csv.reader(f)): if not row or row[0].startswith("#"): continue try: g = float(row[1]) except (IndexError, ValueError): if i == 0: continue # header line sys.exit("bad line %d: %r" % (i + 1, row)) if not 0 <= g <= 1: sys.exit("grade out of 0..1 on line %d" % (i + 1)) by.setdefault(row[0].strip(), []).append(g) if any(len(v) < 2 for v in by.values()): sys.exit("every task needs 2+ judge runs to measure judge noise") out = [] for v in by.values(): m = sum(v) / len(v) w = sum((x - m) ** 2 for x in v) / (len(v) - 1) out.append((m, w, len(v))) return out, "repeat judge runs"def main(): if len(sys.argv) < 2: sys.exit(__doc__.splitlines()[1]) try: items, basis = load(sys.argv[1]) except OSError as e: sys.exit("cannot read %s: %s" % (sys.argv[1], e.strerror)) if not all(k.isdigit() and int(k) > 0 for k in sys.argv[2:]): sys.exit("K must be a positive whole number") ks = [int(k) for k in sys.argv[2:]] or [1, 2, 3, 5, 10] n = len(items) if n < 2: sys.exit("need at least 2 tasks") mean = sum(m for m, _, _ in items) / n vj = sum(w for _, w, _ in items) / n # judge: spread within a task vm = sum((m - mean) ** 2 for m, _, _ in items) / (n - 1) # task means still carry judge noise w/runs; take it out vt = max(vm - sum(w / r for _, w, r in items) / n, 0.0) tot = vt + vj if tot == 0: sys.exit("all grades identical: no noise to split") print("tasks %d, score %.1f%%, judge noise from: %s" % (n, 100 * mean, basis)) print("variance from tasks %.4f, from judge %.4f (judge share %.0f%%)" % (vt, vj, 100 * vj / tot)) print(" judge runs | 95% error bar | 1 run matches it with") for k in ks: part = vt + vj / k need = math.ceil(n * tot / part) print(" %10d | +/- %5.2f pts | %d tasks (%+.0f%%)" % (k, 196 * math.sqrt(part / n), need, 100 * (need / n - 1))) print(" %10s | +/- %5.2f pts | the floor: task noise stays" % ("unlimited", 196 * math.sqrt(vt / n))) print("halving the 1-run bar takes 4x tasks (%d); no K gets there" % (4 * n) if vt > tot / 4 else "halving the 1-run bar is possible with more judge runs")if __name__ == "__main__": main()
For a CSV it estimates the judge part from the spread between repeat runs of a task. Bad input stops it with a message and exit 1; no arguments prints the usage line.
Save the script as judgenoise.py, then clone the data next to it:
git clone --depth 1 --filter=blob:none --sparse \ https://github.com/tatsu-lab/alpaca_eval.gitcd alpaca_evalgit sparse-checkout set results/gpt-4o-2024-05-13R=results/gpt-4o-2024-05-13/weighted_alpaca_eval_gpt4_turbopython3 ../judgenoise.py $R/annotations.json
tasks 805, score 51.3%, judge noise from: judge token oddsvariance from tasks 0.1740, from judge 0.0761 (judge share 30%) judge runs | 95% error bar | 1 run matches it with 1 | +/- 3.45 pts | 805 tasks (+0%) 2 | +/- 3.18 pts | 950 tasks (+18%) 3 | +/- 3.08 pts | 1010 tasks (+25%) 5 | +/- 3.00 pts | 1065 tasks (+32%) 10 | +/- 2.94 pts | 1109 tasks (+38%) unlimited | +/- 2.88 pts | the floor: task noise stayshalving the 1-run bar takes 4x tasks (3220); no K gets there
GPT-4o wins 51.3% against GPT-4 Turbo’s answers. For a score built from one sampled verdict per question, 70% of the variance comes from which questions were picked and 30% from the judge.
One verdict per answer gives a 95% error bar of ±3.45 points. Five verdicts per answer would give ±3.00. That is 4,025 judge calls instead of 805, for a 13% smaller bar.
The right-hand column is the comparison that matters: one verdict per answer on 1,065 questions gives the same ±3.00. That is 260 more test cases to write, run and grade, against 3,220 extra judge calls.
The unlimited row is the floor. Even infinite re-grading stops at ±2.88, because task noise stays. Halving the one-run bar takes four times the questions, and no number of judge runs gets there.
The floor row matches a number AlpacaEval already publishes. Its leaderboard lists GPT-4o’s win rate as 51.33 with a standard error of 1.47, and 1.47 × 1.96 is the ±2.88 in the floor row.
That is because AlpacaEval’s “weighted” judge never uses a sampled verdict. It reads the judge’s probability that the model’s answer is better and uses that as the score. That gets the whole judge-noise cut for the price of one call. A temperature-0 judge also removes the noise but changes the score: it rounds each question to its likelier verdict, giving 53.7% for GPT-4o, not 51.3%.
If your judge API returns token log-probabilities, that is the one setting to change: ask for logprobs on the verdict token and average the probability, not a sampled “yes” or “no”.
The judge share was 30% for GPT-4o and 28% for Claude 3.5 Sonnet (sparse-checkout results/claude-3-5-sonnet-20240620). A different judge, prompt or task can be noisier, so measure the split instead of assuming it.
To check your own judge, grade each answer at least twice and save one line per run as mine.csv next to the script:
task,graderefund-email,1refund-email,1refund-email,0plot-summary,0plot-summary,0plot-summary,0sql-fix,1sql-fix,1sql-fix,1tone-rewrite,1tone-rewrite,0tone-rewrite,1
python3 judgenoise.py mine.csv 1 3 5
tasks 4, score 58.3%, judge noise from: repeat judge runsvariance from tasks 0.1204, from judge 0.1667 (judge share 58%) judge runs | 95% error bar | 1 run matches it with 1 | +/- 52.50 pts | 4 tasks (+0%) 3 | +/- 41.10 pts | 7 tasks (+75%) 5 | +/- 38.42 pts | 8 tasks (+100%) unlimited | +/- 34.00 pts | the floor: task noise stayshalving the 1-run bar takes 4x tasks (16); no K gets there
Four tasks is a toy, so the bars are huge, but the split is the point: here the judge is 58% of the noise, and five runs match doubling the task count.
The judge noise here is the GPT-4 Turbo judge’s token odds on one verdict, with each question’s answers shown in one order. A judge that reasons first, or a re-run that reshuffles the order, adds noise this data cannot show. Seven GPT-4o questions had identical answers and were never judged; counting them as coin flips barely moves the numbers. The talk also means resampling the model’s answers; with one answer per question, that noise sits inside the task part. The leaderboard’s headline score is length-controlled (57.5 for GPT-4o); this uses the raw win rate. Both are older, familiar entries.
Reproduce it: judgenoise.py prints every number in sections 05 and 08 from the two annotations.json files and the CSV above; section 06's figures are arithmetic on that table. The 51.33, 1.47, 53.7 and 57.5 are in the repo's leaderboard CSV under src/alpaca_eval/leaderboards/data_AlpacaEval_2/. The talk's content comes from the video.
With a judge that samples one verdict, re-grading every answer 5 times would cut GPT-4o’s error bar from ±3.45 to ±3.00 points, which 32% more questions also does. Using the judge’s token probability gets the full judge-noise cut in one call. Measure your own split first: the bigger the judge’s share of the noise, the more re-running is worth.
How many times does your eval pipeline grade each answer, and have you checked whether it needs to?
If this saves you a bill, a clap helps others find it.
Follow me for daily LLM deep-dives.
A Netflix Scientist Says Resample Your LLM Judge. 5 Runs Cut the Error Bar 13% was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.