A Netflix Scientist Says Resample Your LLM Judge. 5 Runs Cut the Error Bar 13%

Netflix staff research scientist Cameron Wolfe presented a framework for evaluating AI on tasks without a single correct answer, arguing that LLM judges are random and that resampling verdicts shrinks the error bar. Applied to AlpacaEval 2.0's GPT-4 Turbo judge across 805 public questions, five verdicts per answer cut the error bar from ±3.45 points to ±3.00 points, a 13% reduction, while adding 32% more questions achieves the same result without the 5x judge bill. Wolfe's talk, "How Netflix Evaluates AI When There's No Right Answer," was posted by the AI Engineer channel on November 5 and covers power analysis, error bars and resampling.

If AlpacaEval’s GPT-4 Turbo judge used one sampled verdict per answer, GPT-4o’s error bar would be ±3.45 points. Five verdicts make it ±3.00. So do 32% more questions. Grading every answer 5 times with your LLM judge costs 5x the judge bill. On 805 public AlpacaEval questions, the judge’s own odds show it would shrink the error bar by only 13%. The idea comes from Netflix staff research scientist Cameron Wolfe. After this article you can run one script on your judge’s grades and see whether to pay for extra judge runs or more test cases. Cameron Wolfe, who also writes the Deep Learning Focus newsletter, gave the talk “How Netflix Evaluates AI When There’s No Right Answer” https://www.youtube.com/watch?v=zLc8zeGeNaY , which the AI Engineer channel posted on November 5. It covers tasks like creative writing, where pass or fail is a judgment call. His framework has three parts: His main thread is noise: both the model and its LLM judge are random. His fixes are power analysis to size the eval, error bars on every number, and resampling, meaning multiple rollouts from the model or the judge. Below, I work out what judge resampling buys from a judge whose odds are public. You need a judge that graded many questions and said how sure it was. AlpacaEval 2.0 publishes that. Its GPT-4 Turbo judge compares each model’s answer with GPT-4 Turbo’s own answer on 805 questions, and the repo stores the judge’s win probability for every question, read from its token odds. That probability is what makes the test possible. The judge’s config asks for one token at temperature 1, so if it says 0.7 on a question, one sampled verdict is literally a coin that lands “win” 70% of the time. Its spread, 0.7 × 0.3, is the judge’s own noise on that question. The spread of the probabilities across questions is the noise from which questions you happened to pick. Your eval score is an average over questions, so its error bar has two sources: +--------+--------------------+-----------------+| source | what it is | cure |+========+====================+=================+| tasks | which questions | more questions || judge | a coin per verdict | more judge runs |+--------+--------------------+-----------------+ Grading each answer K times divides only the judge part by K. The task part does not move. judgenoise.py is standard-library Python 3.8+. It reads either an AlpacaEval annotations.json or a CSV of your own grades, splits the variance into those two parts, and prints the error bar for each number of judge runs: bash /usr/bin/env python3"""judgenoise.py - how much of your eval's error bar is the judge?Usage: judgenoise.py FILE K ... FILE is an AlpacaEval annotations.json preference 1..2 or a CSV oftask,grade lines grade 0..1; repeat a task once per judge run .K values: judge runs per answer to price default 1 2 3 5 10 .Stdlib only, Python 3.8+."""import csv, json, math, sysdef load path : returns task mean, judge variance, runs behind the mean , basis if path.endswith ".json" : rows = json.load open path AlpacaEval: preference 1 = baseline wins, 2 = model wins ps = float r "preference" - 1 for r in rows if r.get "preference" not in None, "" the judge's own odds: one sampled verdict has variance p 1-p return p, p 1 - p , math.inf for p in ps , "judge token odds" by = {} with open path, newline="" as f: for i, row in enumerate csv.reader f : if not row or row 0 .startswith " " : continue try: g = float row 1 except IndexError, ValueError : if i == 0: continue header line sys.exit "bad line %d: %r" % i + 1, row if not 0 <= g <= 1: sys.exit "grade out of 0..1 on line %d" % i + 1 by.setdefault row 0 .strip , .append g if any len v < 2 for v in by.values : sys.exit "every task needs 2+ judge runs to measure judge noise" out = for v in by.values : m = sum v / len v w = sum x - m 2 for x in v / len v - 1 out.append m, w, len v return out, "repeat judge runs"def main : if len sys.argv < 2: sys.exit doc .splitlines 1 try: items, basis = load sys.argv 1 except OSError as e: sys.exit "cannot read %s: %s" % sys.argv 1 , e.strerror if not all k.isdigit and int k 0 for k in sys.argv 2: : sys.exit "K must be a positive whole number" ks = int k for k in sys.argv 2: or 1, 2, 3, 5, 10 n = len items if n < 2: sys.exit "need at least 2 tasks" mean = sum m for m, , in items / n vj = sum w for , w, in items / n judge: spread within a task vm = sum m - mean 2 for m, , in items / n - 1 task means still carry judge noise w/runs; take it out vt = max vm - sum w / r for , w, r in items / n, 0.0 tot = vt + vj if tot == 0: sys.exit "all grades identical: no noise to split" print "tasks %d, score %.1f%%, judge noise from: %s" % n, 100 mean, basis print "variance from tasks %.4f, from judge %.4f judge share %.0f%% " % vt, vj, 100 vj / tot print " judge runs | 95% error bar | 1 run matches it with" for k in ks: part = vt + vj / k need = math.ceil n tot / part print " %10d | +/- %5.2f pts | %d tasks %+.0f%% " % k, 196 math.sqrt part / n , need, 100 need / n - 1 print " %10s | +/- %5.2f pts | the floor: task noise stays" % "unlimited", 196 math.sqrt vt / n print "halving the 1-run bar takes 4x tasks %d ; no K gets there" % 4 n if vt tot / 4 else "halving the 1-run bar is possible with more judge runs" if name == " main ": main For a CSV it estimates the judge part from the spread between repeat runs of a task. Bad input stops it with a message and exit 1; no arguments prints the usage line. Save the script as judgenoise.py, then clone the data next to it: git clone --depth 1 --filter=blob:none --sparse \ https://github.com/tatsu-lab/alpaca eval.gitcd alpaca evalgit sparse-checkout set results/gpt-4o-2024-05-13R=results/gpt-4o-2024-05-13/weighted alpaca eval gpt4 turbopython3 ../judgenoise.py $R/annotations.json tasks 805, score 51.3%, judge noise from: judge token oddsvariance from tasks 0.1740, from judge 0.0761 judge share 30% judge runs | 95% error bar | 1 run matches it with 1 | +/- 3.45 pts | 805 tasks +0% 2 | +/- 3.18 pts | 950 tasks +18% 3 | +/- 3.08 pts | 1010 tasks +25% 5 | +/- 3.00 pts | 1065 tasks +32% 10 | +/- 2.94 pts | 1109 tasks +38% unlimited | +/- 2.88 pts | the floor: task noise stayshalving the 1-run bar takes 4x tasks 3220 ; no K gets there GPT-4o wins 51.3% against GPT-4 Turbo’s answers. For a score built from one sampled verdict per question, 70% of the variance comes from which questions were picked and 30% from the judge. One verdict per answer gives a 95% error bar of ±3.45 points. Five verdicts per answer would give ±3.00. That is 4,025 judge calls instead of 805, for a 13% smaller bar. The right-hand column is the comparison that matters: one verdict per answer on 1,065 questions gives the same ±3.00. That is 260 more test cases to write, run and grade, against 3,220 extra judge calls. The unlimited row is the floor. Even infinite re-grading stops at ±2.88, because task noise stays. Halving the one-run bar takes four times the questions, and no number of judge runs gets there. The floor row matches a number AlpacaEval already publishes. Its leaderboard lists GPT-4o’s win rate as 51.33 with a standard error of 1.47, and 1.47 × 1.96 is the ±2.88 in the floor row. That is because AlpacaEval’s “weighted” judge never uses a sampled verdict. It reads the judge’s probability that the model’s answer is better and uses that as the score. That gets the whole judge-noise cut for the price of one call. A temperature-0 judge also removes the noise but changes the score: it rounds each question to its likelier verdict, giving 53.7% for GPT-4o, not 51.3%. If your judge API returns token log-probabilities, that is the one setting to change: ask for logprobs on the verdict token and average the probability, not a sampled “yes” or “no”. The judge share was 30% for GPT-4o and 28% for Claude 3.5 Sonnet sparse-checkout results/claude-3-5-sonnet-20240620 . A different judge, prompt or task can be noisier, so measure the split instead of assuming it. To check your own judge, grade each answer at least twice and save one line per run as mine.csv next to the script: task,graderefund-email,1refund-email,1refund-email,0plot-summary,0plot-summary,0plot-summary,0sql-fix,1sql-fix,1sql-fix,1tone-rewrite,1tone-rewrite,0tone-rewrite,1 python3 judgenoise.py mine.csv 1 3 5 tasks 4, score 58.3%, judge noise from: repeat judge runsvariance from tasks 0.1204, from judge 0.1667 judge share 58% judge runs | 95% error bar | 1 run matches it with 1 | +/- 52.50 pts | 4 tasks +0% 3 | +/- 41.10 pts | 7 tasks +75% 5 | +/- 38.42 pts | 8 tasks +100% unlimited | +/- 34.00 pts | the floor: task noise stayshalving the 1-run bar takes 4x tasks 16 ; no K gets there Four tasks is a toy, so the bars are huge, but the split is the point: here the judge is 58% of the noise, and five runs match doubling the task count. The judge noise here is the GPT-4 Turbo judge’s token odds on one verdict, with each question’s answers shown in one order. A judge that reasons first, or a re-run that reshuffles the order, adds noise this data cannot show. Seven GPT-4o questions had identical answers and were never judged; counting them as coin flips barely moves the numbers. The talk also means resampling the model’s answers; with one answer per question, that noise sits inside the task part. The leaderboard’s headline score is length-controlled 57.5 for GPT-4o ; this uses the raw win rate. Both are older, familiar entries. Reproduce it: judgenoise.py prints every number in sections 05 and 08 from the two annotations.json files and the CSV above; section 06's figures are arithmetic on that table. The 51.33, 1.47, 53.7 and 57.5 are in the repo's leaderboard CSV under src/alpaca eval/leaderboards/data AlpacaEval 2/. The talk's content comes from the video. With a judge that samples one verdict, re-grading every answer 5 times would cut GPT-4o’s error bar from ±3.45 to ±3.00 points, which 32% more questions also does. Using the judge’s token probability gets the full judge-noise cut in one call. Measure your own split first: the bigger the judge’s share of the noise, the more re-running is worth. How many times does your eval pipeline grade each answer, and have you checked whether it needs to? If this saves you a bill, a clap helps others find it. Follow me for daily LLM deep-dives. A Netflix Scientist Says Resample Your LLM Judge. 5 Runs Cut the Error Bar 13% https://pub.towardsai.net/a-netflix-scientist-says-resample-your-llm-judge-5-runs-cut-the-error-bar-13-f07c836de085 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.