{"slug": "llm-as-a-judge-what-it-is-how-accurate-it-is-and-how-to-use-it", "title": "LLM-as-a-judge: what it is, how accurate it is, and how to use it", "summary": "LLM-as-a-judge, an evaluation method in which one large language model grades another model's output against plain-language criteria, agreed with human experts on 85% of non-tied comparisons on MT-Bench in the paper that named the method (Zheng et al., 2023), exceeding the 81% rate at which the human experts agreed with each other. The same research documents predictable judge biases toward the answer it reads first, toward longer answers, and toward its own phrasing, and G-Eval, which has the judge write out evaluation steps before scoring, reached a Spearman correlation of 0.514 with human ratings on summarization (Liu et al., 2023). The guide recommends using judges only for what a rule cannot check, controlling for their biases, and validating them against a team's own labels before a judge gates a release.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# LLM-as-a-judge: what it is, how accurate it is, and how to use it\n\nLLM-as-a-judge uses one model to grade another's output against written criteria. How it works, how accurate it is, its biases, and when not to use it.\n\nLLM-as-a-judge is an evaluation method in which one large language model grades the output of another against criteria written in plain language. You give the judge the input, the response (or two responses to compare), and a criterion, and it returns a score, a pass/fail label or a preference, usually with a one-line reason. It is how teams score open-ended text, such as answers, summaries, support replies and refusals, at a volume no human review team can match.\n\nIt works well enough to be the default for that job. In the paper that named the method, GPT-4\nacting as a judge agreed with human experts on 85% of non-tied comparisons on MT-Bench, more often\nthan the human experts agreed with each other, at 81%\n([Zheng et al., 2023 ↗](https://arxiv.org/abs/2306.05685)). It is also biased in predictable ways:\ntoward the answer it reads first, toward longer answers, and toward its own phrasing. And agreeing\nwith humans is not the same as measuring the right thing. So use a judge for what a rule cannot\ncheck, control for its biases, and validate it against your own labels before it gates a release.\n\nThe rest of this guide covers how a judge works, the three judge types, how accurate judges are, the four biases, how to write a judge prompt, and where a judge should never be the only evaluator.\n\n## ## How LLM-as-a-judge works\n\nA judge is a prompt sent to a model, usually a different one from the model being evaluated. The prompt carries four things:\n\n- +**The task input:** the user message or prompt the evaluated model saw.\n- +**The output under evaluation,** or two outputs when the judge compares.\n- +**The criterion:** one question in plain language, such as \"Does the reply answer the\ncustomer's question without promising a refund?\"\n- +**A reference answer,** optionally, when the task has a known correct answer.\n\nThe judge returns something your code can parse: a number, a label from a fixed set, or A, B or\ntie. Asking it to reason before the verdict usually helps. G-Eval, which has the judge write out\nevaluation steps before it fills in a score, reached a Spearman correlation of 0.514 with human\nratings on summarization, well ahead of the automatic metrics before it\n([Liu et al., 2023 ↗](https://arxiv.org/abs/2303.16634)). The reason also makes a verdict auditable\nwhen someone disputes it.\n\nA pairwise judge prompt looks like this:\n\n```\nYou are comparing two responses to the same customer message.\n\nCustomer message:\n{input}\n\nResponse A:\n{output_a}\n\nResponse B:\n{output_b}\n\nCriterion: Which response answers the customer's actual question? A response\nthat answers it directly is better than one that answers a related question\nor asks for information the customer already gave. Ignore length and tone.\n\nIf both answer it equally well, or neither does, the verdict is TIE.\n\nGive your reason in one sentence, then end with exactly one line:\nVERDICT: A, VERDICT: B or VERDICT: TIE.\n```\n\nRun it over every example in an eval set, aggregate the verdicts, and you have a quality measure that moves when the outputs change.\n\n## ## The three types of LLM judge\n\nThe MT-Bench paper defines three setups, and most eval tools still use them.\n\n| Type | The judge sees | It returns | Best for | \n|---|---|---|---|\n| Pairwise comparison | The input and two outputs | A, B or tie | Comparing two models, prompts or versions on the same inputs | \n| Single-answer grading (pointwise) | The input and one output | A score or label | Absolute pass/fail gates, tracking one system over time | \n| Reference-guided grading | The input, one output and a reference answer | A score or label | Tasks with a known answer that outputs may phrase differently | \n\n**Pairwise is the more stable of the two main types.** A comparison carries its own reference\npoint, the other answer, while a 1-to-10 score is anchored on nothing and drifts between calls and\nbetween judge versions. Its weaknesses are position bias and that it says nothing about whether\neither answer is good enough.\n\n**Pointwise grading gives you an absolute number to threshold,** but only if the rubric pins down\nwhat each score means. \"Rate helpfulness from 1 to 10\" does not; \"1 if the reply cites the order\nnumber it looked up, 0 otherwise\" does. Reference-guided grading is pointwise with an answer key,\nand is only as good as the key.\n\nIf the question is \"did this change make things worse?\", as it is for a model upgrade, a prompt edit or a provider switch, judge pairwise against the current version. That comparison is the decision you are making.\n\n## ## How accurate is LLM-as-a-judge?\n\nAccurate enough to replace a lot of human review, and less accurate than its headline agreement numbers suggest.\n\nThe headline is the 85% from MT-Bench. Count ties as well and the numbers drop to 66% for GPT-4 against humans and 63% for humans against each other, which is the same story at a lower level: a strong judge is about as consistent with experts as experts are with each other.\n\nTwo things keep that number from transferring to your setup:\n\n- +**Judges vary a lot.** In the same paper, a judge gave the same verdict when the two answers\nwere swapped 65% of the time for GPT-4, 46.2% for GPT-3.5 and 23.8% for Claude-v1. The judge\nmodel, the criterion and the domain set the accuracy, so a published figure for one judge on one\nbenchmark says little about yours.\n- +**Agreement is not validity.** An August 2026 study of seven judges across four domains found\nthey held their verdict under edits that should not change it (an invariance score of 0.945) but\nresponded to only 0.319 of minimal edits that should have changed it\n([Chen et al., 2026 ↗](https://arxiv.org/abs/2608.24419) ). The same study reproduced 67.4% of\nMT-Bench's human votes with predictors that look only at surface features. A judge can agree\nwith people by tracking the same surface cues they do, and still miss the change you care about.\n\nThe practical reading: a judge is an instrument you calibrate on your own data, not a property of the model you picked.\n\n## ## LLM-as-a-judge biases\n\nFour biases are well documented. All four are systematic: they push in the same direction on every example, so they do not average out over a larger eval set. They add up.\n\n- +**Position bias.** Judges favor the answer in one slot, usually the first. Fix it by randomizing\nthe order per example, or by running both orders and counting a disagreement as a tie.\n- +**Verbosity bias.** Longer answers read as more thorough. When the MT-Bench authors padded\nanswers by repeating list items that added no information, Claude-v1 and GPT-3.5 preferred the\npadded answer 91.3% of the time and GPT-4 8.7% of the time. Fix it by saying in the criterion\nwhat length means for the task, and measuring length separately with a deterministic check.\n- +**Self-preference.** Judges tend to favor output from their own model family. In MT-Bench, GPT-4\ngave its own answers a 10% higher win rate than humans did and Claude-v1 25%, though the authors\ncall the evidence too limited to be conclusive. Fix it by choosing a judge from a third model\nfamily, and never judge with a model that is itself under comparison.\n- +**Forced choice.** Ask \"which is better?\" with no tie option and the judge picks one even when\nthe two answers are equivalent. Each of those cases becomes a coin flip that adds noise. Fix it\nwith a tie clause that states what equivalence looks like.\n\nThe deep dive on [when to trust an LLM judge](https://www.evalshift.dev/blog/when-to-trust-an-llm-judge) shows how each of\nthese plays out in a paired model-migration test, where self-preference in particular looks\nexactly like the effect you are trying to measure.\n\n## ## How to write an LLM judge prompt\n\nA judge prompt is a spec someone else has to apply. Write it the way you would write one for a colleague.\n\n- +**One criterion per judge.** A prompt that asks about correctness, tone and formatting at once\nreturns a number that says something moved without saying what.\n- +**Name observable behavior, not adjectives.** \"Cites the order id it looked up\" is checkable.\n\"Is helpful\" is a survey question.\n- +**Keep the wording symmetric.** Never tell the judge which response is the new one or the\nbaseline.\n- +**State the tie condition** in the criterion, not as an afterthought.\n- +**Ask for a parseable verdict,** a fixed label or JSON, with a short reason you can read later.\n- +**Use a stronger judge for high-stakes criteria.** Safety and refusal criteria are where a\ncheap judge degrades first.\n- +**Freeze the prompt and the judge model once validated.** Editing either changes the ruler, so\nscores from before and after are no longer comparable.\n\nThe test for a criterion: hand it to a colleague and have them score ten pairs by hand. If a person cannot apply it consistently, neither can the judge.\n\n## ## Validate the judge before you trust it\n\nA judge earns trust on your data, not on a benchmark. The loop is short:\n\n1. +**Label a sample by hand.** Take 30 to 50 examples or pairs from real traffic, labeled by\nwhoever owns the quality bar.\n2. +**Run the judge on the same sample** and count how often it agrees, treating ties honestly.\n3. +**Read every disagreement.** Most trace back to the criterion: ambiguous wording, no tie\ncondition, or a property the criterion never mentioned.\n4. +**Fix the prompt and rerun** until the remaining disagreements are ones you would argue about\nwith a colleague.\n5. +**Run it advisory first.** Let the judge report alongside your other checks for a while before\nit can block a release on its own.\n\n## ## When not to use an LLM judge\n\nIf code can check a property, check it with code. Deterministic checks cost nothing, run instantly, give the same answer every time, and cannot be talked into anything.\n\n- +**Format and schema:** validate JSON against a schema.\n- +**Exact values:** ids, amounts, dates and enum fields.\n- +**Agent tool calls:** which tool, in what order, with which arguments. A judge reading the final\ntext cannot see that an agent said \"I've updated your address\" without ever calling the tool.[Evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) covers those checks.\n- +**Rules that must never break,** such as \"never issue a refund without a lookup first\". Write\nthem as rules on the trace.\n- +**Cost and latency:** measure them.\n\nGive the judge what is left once everything checkable is checked: the questions about the quality of prose. That is a much smaller job than \"which model is better\", and much easier to defend when the result is unwelcome.\n\n## ## LLM-as-a-judge vs human evaluation and metrics\n\n| Method | Cost | Handles open-ended text | Repeatable | Main risk | \n|---|---|---|---|---|\n| Exact match, BLEU, ROUGE | Free | Poorly | Fully | Penalizes a correct answer phrased differently | \n| Embedding similarity | Cents per 1,000 | Partly | Fully, for a fixed model | Measures closeness of meaning, not correctness | \n| LLM judge | About $1 per 1,000 with a small judge | Yes | Mostly, with a frozen prompt and model | The four biases above | \n| Human review | Reviewer hours | Best | No | Does not scale; reviewers drift and tire | \n\nThe judge cost assumes about 2,000 input and 200 output tokens per call on a small model such as\n`gemini-3.1-flash-lite` at $0.25 and $1.50 per million tokens, its\n[October 2026 list price](https://www.evalshift.dev/blog/cheapest-llm-api). A stronger judge costs several times more, which\nis still cheap next to a reviewer.\n\nIn practice you use all four: deterministic checks for everything checkable, a judge for the prose, and people to write the criteria, label the calibration sample and read the cases the judge flagged.\n\n## ## Using LLM-as-a-judge to test a model migration\n\nA model migration is the case where pairwise judging fits best: same inputs, two models, and one\nquestion, whether the new one is worse. This is how the `llm_judge` evaluator in EvalShift works:\n\n- +**Pairwise, one criterion per entry.** Each`llm_judge` entry is its own evaluator with a`criterion_name` , a`criterion_prompt` and a`judge_model` . The judge sees the current and\ncandidate outputs in random order and replies with a winner (A, B or tie) and one short reason.\n- +**Per-example deltas, tested across the suite.** A win scores 1 to 0 and a tie 0.5 each, so each\nexample contributes +1, 0 or -1. The deltas are tested with paired statistics and corrected for\nmultiple comparisons, so a judge's opinion on one example never decides the run.\n- +**Advisory until it is calibrated.**`evalshift init` writes judges with`blocking: false` .\nThey score and appear in the report but cannot fail the verdict. While nothing gates the\nverdict, the recommendations say when a judge has scored at least 20 pairs on every prompt and\nis ready to be switched to blocking.\n- +**A warning on same-family judges.**`doctor` and`validate` warn when a judge model comes from\nthe same provider as either model under comparison.\n- +**Tool-only turns are skipped.** When both sides end in tool calls with no text, there is\nnothing for a judge to read. The turn is skipped rather than scored as a tie, and is left to the\ntool-call evaluators.\n\n```\nevaluators:\n  llm_judge:\n    - criterion_name: answers_the_question\n      criterion_prompt: |\n        Which response answers the customer's actual question? A direct answer\n        is better than one that answers a related question or asks for\n        information the customer already gave. Ignore length and tone. If both\n        answer it equally well, or neither does, answer TIE.\n      judge_model: gemini-3.1-pro-preview\n      blocking: false\n```\n\nWithout an explicit `judge_model`, a criterion runs on `gemini-3.1-flash-lite-preview`, a\ndeliberately cheap default. The [evaluators reference](https://www.evalshift.dev/docs/evaluators#judge) has every field,\nand the [migration test guide](https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship) shows the judge in a\nfull run.\n\n## ## FAQ\n\n**What is LLM-as-a-judge?** An evaluation method in which a large language model grades another\nmodel's output against criteria written in plain language. The judge receives the input, the\noutput or two outputs to compare, and a criterion, and returns a score, a label or a preference,\nusually with a short reason. It is used to score open-ended text at a scale human review cannot\nreach.\n\n**How accurate is LLM-as-a-judge?** In the MT-Bench study, GPT-4 as a judge agreed with human\nexperts on 85% of non-tied comparisons, against 81% agreement between the experts themselves.\nAccuracy varies widely by judge model, criterion and domain, and agreement with humans does not\nprove the judge measures what you intend, so validate a judge on your own labeled sample.\n\n**What are the main biases of LLM judges?** Position bias (favoring the answer in one slot),\nverbosity bias (favoring longer answers), self-preference (favoring output from the judge's own\nmodel family) and forced choice (picking a winner between equivalent answers when no tie is\nallowed). Randomized order, length-aware criteria, a judge from a third model family and an\nexplicit tie clause address each one.\n\n**Is pairwise or pointwise LLM judging better?** Pairwise comparison is more stable, because the\nother answer gives the judge a reference point, and it fits decisions like model upgrades and\nprompt changes. Pointwise grading gives an absolute score you can threshold, but only with a rubric\nthat defines what each score means.\n\n**When should you not use an LLM as a judge?** When code can check the property: schema\nconformance, exact values, agent tool calls and their arguments, hard rules, cost and latency.\nDeterministic checks are free, repeatable and unbiased. Use the judge for the quality of prose\nthat remains once those are checked.\n\n**Which model should be the judge?** One from a different model family than the models being\ncompared, so self-preference cannot favor one side, and a stronger model for high-stakes criteria\nsuch as safety. Pin the judge model and freeze its prompt once validated, so scores stay\ncomparable over time.\n\n## ## Keep reading\n\n- +[When to trust an LLM judge](https://www.evalshift.dev/blog/when-to-trust-an-llm-judge) : the four failure modes in a\npaired migration test, and keeping a criterion advisory until it earns its vote.\n- +[Evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) : the deterministic checks that\nshould run before any judge.\n- +[How to test an LLM model migration before you ship it](https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship) :\nthe paired run a judge is one input to.\n- +[Evaluators](https://www.evalshift.dev/docs/evaluators) : every evaluator, every field, every default.", "url": "https://wpnews.pro/news/llm-as-a-judge-what-it-is-how-accurate-it-is-and-how-to-use-it", "canonical_source": "https://www.evalshift.dev/blog/llm-as-a-judge", "published_at": "2026-10-27 00:00:00+00:00", "updated_at": "2026-10-27 22:13:52.631002+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools", "artificial-intelligence", "mlops"], "entities": ["MT-Bench", "GPT-4", "Zheng et al., 2023", "G-Eval", "Liu et al., 2023"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-as-a-judge-what-it-is-how-accurate-it-is-and-how-to-use-it", "markdown": "https://wpnews.pro/news/llm-as-a-judge-what-it-is-how-accurate-it-is-and-how-to-use-it.md", "text": "https://wpnews.pro/news/llm-as-a-judge-what-it-is-how-accurate-it-is-and-how-to-use-it.txt", "jsonld": "https://wpnews.pro/news/llm-as-a-judge-what-it-is-how-accurate-it-is-and-how-to-use-it.jsonld"}}