# An LLM judge is a biased instrument, not a measurement > Source: > Published: 2026-07-22 19:27:15+00:00 Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Variant B won. Same model, same judge, same test set. The only thing that changed was the order the two answers were pasted into the judge prompt. That is not a flake. It is position bias, and it is one of three biases that a careful paper documented three years ago. Lianmin Zheng and coauthors measured them in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023, arXiv:2306.05685). Their headline number is genuinely encouraging: a strong judge like GPT-4 agreed with human preferences more than 80 percent of the time, about the same rate at which two humans agree with each other. Their second finding is the one people quote less often. The same judge shows position bias, verbosity bias, and self-enhancement bias. If you report an eval score without accounting for those, you are reporting a property of the judge as if it were a property of your model. There is a habit, imported from good statistics, of assuming that if a measurement is noisy you take more readings and the noise shrinks. That works for random error. The standard error of a mean falls with the square root of the sample size, so a thousand judge calls do give you a tighter estimate than ten. The catch is that position, verbosity, and self-preference are not random error. They are systematic. Run the judge a thousand times with the wordier answer always in slot B, and the wordier answer wins a little more often than it should, every time. The bias sits in the expected value, not the variance. More samples estimate the biased number more precisely, so you end up more sure of a number that is off center. Three specific ways this shows up: Position. Many judges prefer whichever answer they read first, or second, depending on the model. If your harness always puts the baseline first and the candidate second, every comparison inherits the same tilt. Verbosity. Judges reward length and apparent thoroughness. A change that only made the output longer can win a bakeoff it did not deserve, and you will ship a prompt that pads. Self-preference. A judge tends to score outputs from its own model family higher. This quietly contaminates any comparison where the judge is also, directly or by family, one of the contestants. I use several of these, and the landscape is more alike than the marketing suggests. Almost all of them ship an LLM-as-judge, and most stop there. DeepEval builds several of its metrics on a judge, G-Eval being a chain-of-thought scorer that asks a model to grade against a rubric. RAGAS scores faithfulness and answer relevancy with a judge as well. OpenAI Evals gives you model-graded templates alongside exact-match checks. Langfuse lets you attach judge evaluators to traced production data, which is useful precisely because it runs on real traffic rather than a frozen set. A smaller group gives you cheap deterministic scorers you can run first, and treat the judge as the expensive second opinion. promptfoo mixes deterministic assertions (equality, regex, JSON-schema) with model-graded ones in the same test. Future AGI's eval SDK takes the same shape, running local metrics with no network call behind one evaluate() function and adding a judge only when you pass augment=True. Braintrust's autoevals ships both heuristic scorers and LLM-judge scorers. Langfuse and Braintrust, like Future AGI, also attach evaluation to tracing rather than being eval-only, so you can score what actually ran. None of that is a knock on any of them, and it is not a ranking. Some of them go further than handing you the instrument. Langfuse's score analytics lines human annotations up against judge scores and reports Cohen's kappa, correlation, and percent agreement, with the interpretation bands printed next to the number. Braintrust puts human and automated scores side by side on the same traces so you can watch how often they agree and refine the scorer when they drift. That is real, and you should use it. It is also not the whole job. A tool can compute how often the judge agrees with your labels. It cannot produce the labels for your task, and it cannot decide how much agreement is enough for the call you are about to make. Those two stay with you no matter which tool you pick. Treat the judge like any other instrument you did not build. Calibrate it, then use it. Calibrate against a human-labeled sample. Label a few hundred examples by hand, run the judge on the same set, and compute agreement. Cohen's kappa is the honest version because it discounts the agreement you would get by chance. The calculation is one function call, and as noted a couple of platforms will run it for you; what none of them can outsource is labeling your data and picking the threshold you will accept: ``` python from sklearn.metrics import cohen_kappa_score # human_labels and judge_labels are lists of the same labels, e.g. "pass"/"fail" or "A"/"B" kappa = cohen_kappa_score(human_labels, judge_labels) # < 0.41 is Fair-to-Slight on the Landis and Koch bands (the same ones Langfuse prints); # there the judge is not yet a stand-in for your reviewers on this task ``` Swap positions and check consistency. If the winner flips when you swap the order, the score is order-dependent, not a property of the answers. This is a few lines around whatever judge call you already have: ``` python def judge(question, answer_a, answer_b): """Return 'A' or 'B'. Wrap your own LLM-as-judge call here.""" ... def swap_consistency(question, a, b): first = judge(question, a, b) # candidate 'a' sits in slot A second = judge(question, b, a) # candidate 'a' sits in slot B a_wins_first = first == "A" a_wins_second = second == "B" return a_wins_first == a_wins_second # True if the verdict held after the swap # Average this over your eval set. On my sets I start to distrust a judge below # about 0.9 here: at that point it is grading position as much as content, # so score both orders and average them. ``` Triage with something cheap. A local metric or a deterministic check can resolve the clear cases for free, which shrinks how many times you lean on the biased judge at all. Reserve the judge for the genuinely uncertain tail. This is the one place the tool choice helps, because the tools that ship local scorers make the triage a config flag rather than a rewrite. Report agreement next to the score. "94 percent pass rate" means little on its own. "94 percent pass rate, judge-human kappa 0.71 on a 300-example calibration set, scored in both orders" is a number a reviewer can trust, and it is the same amount of work to print. None of this makes the judge unbiased. It makes the bias visible and gives you a number for how large it is, which is the most you can honestly claim for an instrument you did not build. GPT-4 agrees with humans more than 80 percent of the time. Is that not good enough? It depends entirely on what the disagreeing 20 percent correlates with. If those cases were random, you could average them away. They are not random. They line up with length and position, so the error has structure, and structure survives averaging. The aggregate agreement can look fine while every close call tilts the same direction. Pairwise or pointwise scoring? Pairwise (A versus B) is usually more stable than asking for an absolute 1-to-10 score, because relative judgments are easier for a model to make consistently. But pairwise is exactly where position bias lives, so if you go pairwise you have to score both orders and average. Pointwise avoids position bias and trades it for scale drift, where the judge's notion of "7 out of 10" wanders between runs. Do I need human labels forever? No, you need a calibration sample. Label a few hundred examples once, measure judge-human agreement, and recheck when something changes: a new judge model, a new prompt, a shift in the kind of inputs you see. The labels are a periodic audit, not a standing tax. Will a bigger judge model fix this? It raises raw agreement, which is why people reach for the strongest model they can afford. It can also make self-preference worse, because a stronger judge is often from the same family as the strong models you are testing. The cheap defense is to pick a judge from a different family than the models under test. The thing I do not have a clean answer to is drift. You calibrate against human labels in July, the provider silently updates the hosted judge model in September, and your kappa is now describing a different instrument than the one you validated. Nobody I know version-pins their judge the way they pin a training dependency, and most hosted judges do not expose a stable version to pin to. So how often is often enough to recalibrate, and should a judge model be treated as a pinned dependency with its own changelog? I have not seen a convention settle, and I would take pointers from anyone who has one that works.