{"slug": "the-complete-guide-to-llm-evaluation", "title": "The Complete Guide to LLM Evaluation", "summary": "A guide to LLM evaluation outlines a systematic approach for grading LLM application outputs, splitting scorers into three categories: code scorers for deterministic checks like JSON parsing and length limits, reference-based scorers such as exact match, F1 and BERTScore for questions with one right answer, and LLM judges for criteria like tone and groundedness that rules cannot express. The guide recommends starting with code scorers wherever a rule works because they cost nothing and return the same result every time, and adding a judge only for criteria rules cannot capture, with judge calibration measured against human labels using Cohen's kappa. It also advises using evaluation to guide iteration and binary tests to gate releases, rather than blocking deployment on an arbitrary faithfulness threshold such as 0.7.", "body_md": "In this article, you will learn how to evaluate LLM applications systematically, from choosing scorers and designing reliable LLM judges to evaluating RAG systems and agents across the release cycle.\n\nTopics we will cover include:\n\n- How code, reference-based, and LLM-judge scorers differ, and how to write, score, and calibrate an LLM judge with rubrics, QAG, G-Eval, pairwise comparison, and Cohen’s kappa.\n- How to score RAG systems and agents component by component, so you can tell which part failed.\n- How to run offline evals before release and online evals on live traffic.\n\n## Introduction\n\nYou test most LLM apps by reading a few outputs and deciding they look fine. A prompt edit, a model swap, or a retrieval change then breaks something you never checked. And by then you can’t quite tell which change caused it.\n\n**LLM evaluation** catches these breaks before users do. You build test cases, score each output against written criteria, and compare scores before and after every change. It is the regression-testing loop from regular code, adapted for outputs that differ on every run.\n\nThis article covers:\n\n- How code, reference-based, and LLM-judge scorers work, and when to use each\n- How LLM-as-a-judge works: rubric design, QAG, G-Eval, and pairwise comparison\n- How to measure judge bias against human labels\n- How to evaluate RAG systems and agents, and run evals before and after release\n\nWe start with why spot-checking breaks down.\n\n## Seeing Why Spot-Checking Breaks Down\n\nLLM outputs vary between runs, and “correct” depends on context. A unit test asserts equality, but “You can return shoes within 30 days” and “Returns are accepted for a month after delivery” are both right, and no string comparison knows it.\n\nSo here are two ideas to fill the gap:\n\n- Evaluation measures relative quality across fuzzy criteria: is version B better than version A, and on which inputs?\n- Testing gates a release with binary checks, such as valid JSON or a banned phrase never appearing.\n\nIf you block deployment on a faithfulness score of 0.7, you will argue about an arbitrary threshold every week. Use evaluation to guide iteration and tests to protect the release.\n\nEvery eval has three parts: a dataset of inputs, scorers that grade outputs, and a record of scores over time. The rest of this article builds on these.\n\n## Choosing Scorers\n\nA scorer takes a test case (input, output, and sometimes context or a reference) and returns a verdict or a number. There are three categories, as shown below:\n\n| Scorer | How it grades | Best for | Example | \n|---|---|---|---|\n| Code | Deterministic logic | Format, length, required strings | JSON parses, under 120 words | \n| Reference-based | Compares output to a known answer | Questions with one right answer | Exact match, F1, embedding similarity | \n| LLM judge | A model applies your rubric | Tone, groundedness, helpfulness | “Is every claim supported by the context?” | \n\n[Overlap metrics such as BLEU and ROUGE count shared n-grams with a reference](https://deepchecks.com/advanced-techniques-evaluating-llm-text-summarization/), so a correct paraphrase scores low. Embedding-based scores such as [BERTScore](https://arxiv.org/abs/1904.09675) handle paraphrase through [cosine similarity](https://www.ibm.com/think/topics/cosine-similarity) between token embeddings, but they need a reference and cannot judge tone or whether a claim is supported by a source.\n\nStart with code scorers wherever a rule works. They cost nothing and return the same result every time. Add a judge only for the criteria that rules cannot express.\n\n## Understanding LLM-as-a-Judge\n\nAn [LLM judge](https://www.datadoghq.com/knowledge-center/llm-as-a-judge/) is a model prompted to grade another model’s output. Its job is narrower than generation: it reads a rubric and a candidate answer, then emits a verdict. Three setups cover most uses:\n\n- Pointwise: the judge scores one output against a rubric, as a label (PASS or FAIL) or a number on a scale.\n- Pairwise: the judge sees two outputs for the same input and picks the better one. This is the usual setup for comparing two prompts or two models.\n- Reference-guided: the judge also receives a gold answer and checks the output against it. This helps on factual questions where the judge might not know the truth itself.\n\n### Writing the rubric prompt\n\nThe rubric is the whole product. A usable judge prompt has four parts: a definition of the criterion, the inputs the judge may use, a procedure, and a strict output format. Vague criteria like “is the answer good” produce vague scores. Compare that with this faithfulness judge for a policy-answering bot:\n\n``` python\nimport json\n\nRUBRIC = \"\"\"You are grading a customer-support answer for faithfulness.\n\nPolicy: {context}\nQuestion: {input}\nAnswer: {answer}\n\nSteps:\n1. List each factual claim in the Answer.\n2. For each claim, decide whether the Policy supports it.\n3. Verdict: PASS if every claim is supported, otherwise FAIL.\n\nRespond with JSON only:\n{{\"claims\": [{{\"text\": \"...\", \"supported\": true}}], \"verdict\": \"PASS\"}}\"\"\"\n\ndef judge(case, answer):\n    response = call_llm(RUBRIC.format(answer=answer, **case), temperature=0)\n    return json.loads(response)\n\n12345678910111213141516171819\n\nimport json RUBRIC = \"\"\"You are grading a customer-support answer for faithfulness. Policy: {context}Question: {input}Answer: {answer} Steps:1. List each factual claim in the Answer.2. For each claim, decide whether the Policy supports it.3. Verdict: PASS if every claim is supported, otherwise FAIL. Respond with JSON only:{{\"claims\": [{{\"text\": \"...\", \"supported\": true}}], \"verdict\": \"PASS\"}}\"\"\" def judge(case, answer):    response = call_llm(RUBRIC.format(answer=answer, **case), temperature=0)    return json.loads(response)\n```\n\n`call_llm` is your own function that sends a prompt to a model and returns text. Three design choices in this prompt matter. The reasoning (the `claims` list) comes before the verdict, so the verdict is conditioned on evidence.\n\nThe output is structured JSON, so you can parse it and log the reasoning when a score looks wrong. And temperature is 0, which reduces sampling noise. It does not remove it, because floating-point nondeterminism and provider-side batching still produce occasional flips on borderline cases. For criteria where flips matter, run the judge an odd number of times and take the majority verdict.\n\nThe claims list also gives you a score without asking the model for a number.\n\n### Scoring with QAG\n\nAsking an LLM “rate this from 1 to 10” gives unstable results, because the model has no stable internal scale. [Question-answer generation (QAG)](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/qag-score) avoids this. The judge answers closed yes/no questions, and your code does the arithmetic. In the example above, each claim is a yes/no question (“does the policy support this?”), and faithfulness is:\n\n```\nscore = sum(c[\"supported\"] for c in verdict[\"claims\"]) / len(verdict[\"claims\"])\n\n1\n\nscore = sum(c[\"supported\"] for c in verdict[\"claims\"]) / len(verdict[\"claims\"])\n```\n\nAn answer with four claims and three supported scores 0.75. The model only classifies. Classification is the task LLMs do most consistently, and the score is reproducible given the same claim verdicts. Most RAG faithfulness metrics work this way.\n\n### Scoring with G-Eval\n\n[G-Eval](https://arxiv.org/abs/2303.16634) targets subjective criteria like coherence or helpfulness, where there are no discrete claims to count. It works in two stages:\n\nFirst, the framework uses an automatic chain-of-thought (Auto-CoT) prompt to decompose a high-level evaluation criterion into a step-by-step evaluation pipeline.\n\nSecond, a form-filling generation prompt combines these steps with the target text, generating a score from 1 to 5 ([or 0 to 10, depending on the framework](https://deepeval.com/docs/metrics-llm-evals)), calculated via a probability-weighted summation of output tokens to ensure stability.\n\nThe distinctive part is how the final number is computed. G-Eval reads the probabilities of the score tokens “1” through “5” and returns the expected value, the probability-weighted average. A response where the model splits 60/40 between 4 and 5 scores 4.4. This gives finer resolution and reduces the model’s habit of collapsing to a favorite number. It requires token log-probabilities, which not every API exposes. Without them, you can approximate by sampling several judgments and averaging.\n\nUse QAG when the criterion decomposes into checkable facts. Use G-Eval when it does not.\n\n### Running pairwise comparisons\n\nPairwise judges answer “is A or B better?” and avoid absolute scales. They are sensitive to answer order, so run each comparison twice with the positions swapped and keep only the verdicts that agree:\n\n``` python\ndef prefer(case, a, b):\n    first = pairwise(case, a, b)    # returns \"A\" or \"B\"\n    second = pairwise(case, b, a)\n    if first == \"A\" and second == \"B\":\n        return \"a\"\n    if first == \"B\" and second == \"A\":\n        return \"b\"\n    return \"tie\"\n\n12345678\n\ndef prefer(case, a, b):    first = pairwise(case, a, b)    # returns \"A\" or \"B\"    second = pairwise(case, b, a)    if first == \"A\" and second == \"B\":        return \"a\"    if first == \"B\" and second == \"A\":        return \"b\"    return \"tie\"\n```\n\nA “tie” here means the judge’s preference followed position, not content. Count those as ties and report the tie rate. A high tie rate tells you the two outputs are hard to separate.\n\n## Making Judges Reliable\n\n[Research on judging LLM outputs](https://arxiv.org/abs/2306.05685) documents a few consistent biases:\n\n- Position bias favors the answer in a particular slot.\n- Verbosity bias favors the longer answer even when it adds nothing.\n- Self-preference bias favors outputs from the judge’s own model family.\n\nMitigations match each one: swap positions, add “do not reward length” and a length-matched test case to your dataset, and use a judge from a different family than the system under test.\n\nThe most important mitigation is calibration against human labels. Label 30 to 50 outputs yourself, run the judge on the same set, and compare. Raw agreement misleads when most cases pass. If 90% of your cases are PASS, a judge that always says PASS agrees with you 90% of the time and tells you nothing. [Cohen’s kappa](https://www.knime.com/blog/cohens-kappa-an-overview) corrects for chance agreement. With scikit-learn, you can compute it like so:\n\n``` python\nfrom sklearn.metrics import cohen_kappa_score\n\nkappa = cohen_kappa_score(human_labels, judge_labels)\n\n123\n\nfrom sklearn.metrics import cohen_kappa_score kappa = cohen_kappa_score(human_labels, judge_labels)\n```\n\nA kappa of 0 means chance-level agreement, and 1 means perfect. Values above roughly 0.6 are usually read as substantial agreement. When kappa is low, read the disagreements, find what the judge misreads, and change the rubric. Anchoring each level of a scale with an example output also helps. Treat the labeled set as a regression test for the judge itself, and rerun it whenever you change the judge prompt or its model.\n\n## Evaluating RAG Systems\n\nA [RAG (retrieval-augmented generation) system](https://cloud.google.com/use-cases/retrieval-augmented-generation) has two components that fail independently. The retriever finds context, and the generator writes an answer from it. Score them separately like so:\n\n- Context precision: are the retrieved chunks relevant, with the best ones ranked first?\n- Context recall: did retrieval find everything the answer needs?\n- Faithfulness: is every claim in the answer supported by the retrieved context?\n- Answer relevance: does the answer address the question that was asked?\n\nBut what happens when faithfulness is high and recall is low? The model stays grounded in what it received, and it received too little. Fix retrieval. High precision with low faithfulness points the other way: good documents, invented claims, so the fix lives in the prompt or the model.\n\n[Reference-free RAG metrics](https://arxiv.org/abs/2309.15217) compute faithfulness and relevance without labeled answers, so they run on live traffic. Recall needs an expected answer, so it stays an offline metric. For hallucination checks with no context at all, [sampling-based methods](https://arxiv.org/abs/2303.08896) generate several answers to one prompt and compare them. Facts the model knows repeat across samples, while invented ones drift. The cost is several calls per answer.\n\n## Evaluating Agents and Conversations\n\n[Agents](https://www.ibm.com/think/topics/ai-agents) add tool calls, planning, and loops. Start with task completion: did the user’s goal get done? A support agent with fluent replies and unresolved tickets fails at its job. When completion drops, three diagnostic scores locate the cause:\n\n- Tool correctness: did the agent pick the right tool? This is usually an exact match against an expected tool list, with no judge involved.\n- Argument correctness: were the parameters well formed and sensible for the input?\n- Trajectory: did it take a reasonable path, without redundant calls or loops?\n\nMulti-turn systems need the same treatment at the thread level. Each turn can look fine while the conversation forgets earlier context or contradicts itself, so give the judge the full history and score the outcome. Component scores sit under the end-to-end score: when completion falls, they show which step broke.\n\n## Running Evals Across the Release Cycle\n\n**Offline evals** run before you ship. Build a set of 25 to 50 cases drawn from representative queries, and cover normal requests, edge cases, adversarial inputs, and off-topic questions the bot should decline. Version the set with your code and refresh it as usage changes. Write release criteria in advance: minimum scores, a maximum regression against the current production version, and every safety check passing. Run the suite on each pull request and report which cases improved and which regressed.\n\n**Online evals** run after you ship. Sample live traffic and score it with reference-free checks (faithfulness, tone, format) next to latency and cost. This catches inputs your dataset never imagined and quality drift when a provider updates a model.\n\nCommon challenges in LLM evaluation include overfitting to test sets, data leakage, and metric gaming. Use holdout sets, keep evaluation data separate, and track diverse metrics to assess quality. Choose metrics based on the use case and system architecture.\n\n## Summary & Further Reading\n\nThe table summarizes what to do at each stage of an eval setup:\n\n| Stage | What to do | Metric or technique | \n|---|---|---|\n| Scorer choice | Use code where a rule works, reference-based scorers for known answers, and a judge for subjective criteria. | Schema or regex checks, exact match, F1, LLM judge | \n| Judge rubric | Define the criterion, inputs, steps, and a strict JSON output with reasoning before the verdict. | Temperature 0, majority vote over an odd number of runs | \n| Judge scoring | Break the criterion into checkable claims, score a subjective one on a scale, or compare two outputs. | QAG (supported claims divided by total), G-Eval (probability-weighted score), pairwise with swapped order | \n| Judge calibration | Label 30 to 50 outputs yourself and compare them with the judge. | Cohen’s kappa, position swaps, a judge from a different model family | \n| RAG | Score the retriever and the generator separately. | Context precision, context recall, faithfulness, answer relevance | \n| Agents | Check task completion first, then diagnose failures. | Tool correctness, argument correctness, trajectory | \n| Offline evals | Run a versioned golden set on every pull request against written release criteria. | 25 to 50 cases, a holdout set, maximum allowed regression | \n| Online evals | Sample live traffic and turn each production failure into a golden case. | Faithfulness, tone, format, latency, cost | \n\nHere are a few resources for further learning:\n\n- [Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)\n- [G-Eval: NLG evaluation using GPT-4 with better human alignment](https://arxiv.org/abs/2303.16634)\n- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)\n- [Ragas: automated evaluation of retrieval augmented generation](https://arxiv.org/abs/2309.15217)\n\nHappy learning!", "url": "https://wpnews.pro/news/the-complete-guide-to-llm-evaluation", "canonical_source": "https://machinelearningmastery.com/the-complete-guide-to-llm-evaluation/", "published_at": "2026-11-13 12:02:40+00:00", "updated_at": "2026-11-13 12:21:56.097812+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-agents", "artificial-intelligence"], "entities": ["BERTScore", "G-Eval", "Cohen's kappa", "Deepchecks", "IBM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-complete-guide-to-llm-evaluation", "markdown": "https://wpnews.pro/news/the-complete-guide-to-llm-evaluation.md", "text": "https://wpnews.pro/news/the-complete-guide-to-llm-evaluation.txt", "jsonld": "https://wpnews.pro/news/the-complete-guide-to-llm-evaluation.jsonld"}}