In this article, you will learn how to evaluate LLM applications systematically, from choosing scorers and designing reliable LLM judges to evaluating RAG systems and agents across the release cycle.
Topics we will cover include:
- How code, reference-based, and LLM-judge scorers differ, and how to write, score, and calibrate an LLM judge with rubrics, QAG, G-Eval, pairwise comparison, and Cohen’s kappa.
- How to score RAG systems and agents component by component, so you can tell which part failed.
- How to run offline evals before release and online evals on live traffic.
Introduction #
You test most LLM apps by reading a few outputs and deciding they look fine. A prompt edit, a model swap, or a retrieval change then breaks something you never checked. And by then you can’t quite tell which change caused it.
LLM evaluation catches these breaks before users do. You build test cases, score each output against written criteria, and compare scores before and after every change. It is the regression-testing loop from regular code, adapted for outputs that differ on every run.
This article covers:
- How code, reference-based, and LLM-judge scorers work, and when to use each
- How LLM-as-a-judge works: rubric design, QAG, G-Eval, and pairwise comparison
- How to measure judge bias against human labels
- How to evaluate RAG systems and agents, and run evals before and after release
We start with why spot-checking breaks down.
Seeing Why Spot-Checking Breaks Down #
LLM outputs vary between runs, and “correct” depends on context. A unit test asserts equality, but “You can return shoes within 30 days” and “Returns are accepted for a month after delivery” are both right, and no string comparison knows it.
So here are two ideas to fill the gap:
- Evaluation measures relative quality across fuzzy criteria: is version B better than version A, and on which inputs?
- Testing gates a release with binary checks, such as valid JSON or a banned phrase never appearing.
If you block deployment on a faithfulness score of 0.7, you will argue about an arbitrary threshold every week. Use evaluation to guide iteration and tests to protect the release.
Every eval has three parts: a dataset of inputs, scorers that grade outputs, and a record of scores over time. The rest of this article builds on these.
Choosing Scorers #
A scorer takes a test case (input, output, and sometimes context or a reference) and returns a verdict or a number. There are three categories, as shown below:
| Scorer | How it grades | Best for | Example |
|---|---|---|---|
| Code | Deterministic logic | Format, length, required strings | JSON parses, under 120 words |
| Reference-based | Compares output to a known answer | Questions with one right answer | Exact match, F1, embedding similarity |
| LLM judge | A model applies your rubric | Tone, groundedness, helpfulness | “Is every claim supported by the context?” |
Overlap metrics such as BLEU and ROUGE count shared n-grams with a reference, so a correct paraphrase scores low. Embedding-based scores such as BERTScore handle paraphrase through cosine similarity between token embeddings, but they need a reference and cannot judge tone or whether a claim is supported by a source.
Start with code scorers wherever a rule works. They cost nothing and return the same result every time. Add a judge only for the criteria that rules cannot express.
Understanding LLM-as-a-Judge #
An LLM judge is a model prompted to grade another model’s output. Its job is narrower than generation: it reads a rubric and a candidate answer, then emits a verdict. Three setups cover most uses:
- Pointwise: the judge scores one output against a rubric, as a label (PASS or FAIL) or a number on a scale.
- Pairwise: the judge sees two outputs for the same input and picks the better one. This is the usual setup for comparing two prompts or two models.
- Reference-guided: the judge also receives a gold answer and checks the output against it. This helps on factual questions where the judge might not know the truth itself.
Writing the rubric prompt
The rubric is the whole product. A usable judge prompt has four parts: a definition of the criterion, the inputs the judge may use, a procedure, and a strict output format. Vague criteria like “is the answer good” produce vague scores. Compare that with this faithfulness judge for a policy-answering bot:
import json
RUBRIC = """You are grading a customer-support answer for faithfulness.
Policy: {context}
Question: {input}
Answer: {answer}
Steps:
1. List each factual claim in the Answer.
2. For each claim, decide whether the Policy supports it.
3. Verdict: PASS if every claim is supported, otherwise FAIL.
Respond with JSON only:
{{"claims": [{{"text": "...", "supported": true}}], "verdict": "PASS"}}"""
def judge(case, answer):
response = call_llm(RUBRIC.format(answer=answer, **case), temperature=0)
return json.loads(response)
12345678910111213141516171819
import json RUBRIC = """You are grading a customer-support answer for faithfulness. Policy: {context}Question: {input}Answer: {answer} Steps:1. List each factual claim in the Answer.2. For each claim, decide whether the Policy supports it.3. Verdict: PASS if every claim is supported, otherwise FAIL. Respond with JSON only:{{"claims": [{{"text": "...", "supported": true}}], "verdict": "PASS"}}""" def judge(case, answer): response = call_llm(RUBRIC.format(answer=answer, **case), temperature=0) return json.loads(response)
call_llm is your own function that sends a prompt to a model and returns text. Three design choices in this prompt matter. The reasoning (the claims list) comes before the verdict, so the verdict is conditioned on evidence.
The output is structured JSON, so you can parse it and log the reasoning when a score looks wrong. And temperature is 0, which reduces sampling noise. It does not remove it, because floating-point nondeterminism and provider-side batching still produce occasional flips on borderline cases. For criteria where flips matter, run the judge an odd number of times and take the majority verdict.
The claims list also gives you a score without asking the model for a number.
Scoring with QAG
Asking an LLM “rate this from 1 to 10” gives unstable results, because the model has no stable internal scale. Question-answer generation (QAG) avoids this. The judge answers closed yes/no questions, and your code does the arithmetic. In the example above, each claim is a yes/no question (“does the policy support this?”), and faithfulness is:
score = sum(c["supported"] for c in verdict["claims"]) / len(verdict["claims"])
1
score = sum(c["supported"] for c in verdict["claims"]) / len(verdict["claims"])
An answer with four claims and three supported scores 0.75. The model only classifies. Classification is the task LLMs do most consistently, and the score is reproducible given the same claim verdicts. Most RAG faithfulness metrics work this way.
Scoring with G-Eval
G-Eval targets subjective criteria like coherence or helpfulness, where there are no discrete claims to count. It works in two stages:
First, the framework uses an automatic chain-of-thought (Auto-CoT) prompt to decompose a high-level evaluation criterion into a step-by-step evaluation pipeline.
Second, a form-filling generation prompt combines these steps with the target text, generating a score from 1 to 5 (or 0 to 10, depending on the framework), calculated via a probability-weighted summation of output tokens to ensure stability.
The distinctive part is how the final number is computed. G-Eval reads the probabilities of the score tokens “1” through “5” and returns the expected value, the probability-weighted average. A response where the model splits 60/40 between 4 and 5 scores 4.4. This gives finer resolution and reduces the model’s habit of collapsing to a favorite number. It requires token log-probabilities, which not every API exposes. Without them, you can approximate by sampling several judgments and averaging.
Use QAG when the criterion decomposes into checkable facts. Use G-Eval when it does not.
Running pairwise comparisons
Pairwise judges answer “is A or B better?” and avoid absolute scales. They are sensitive to answer order, so run each comparison twice with the positions swapped and keep only the verdicts that agree:
def prefer(case, a, b):
first = pairwise(case, a, b) # returns "A" or "B"
second = pairwise(case, b, a)
if first == "A" and second == "B":
return "a"
if first == "B" and second == "A":
return "b"
return "tie"
12345678
def prefer(case, a, b): first = pairwise(case, a, b) # returns "A" or "B" second = pairwise(case, b, a) if first == "A" and second == "B": return "a" if first == "B" and second == "A": return "b" return "tie"
A “tie” here means the judge’s preference followed position, not content. Count those as ties and report the tie rate. A high tie rate tells you the two outputs are hard to separate.
Making Judges Reliable #
Research on judging LLM outputs documents a few consistent biases:
- Position bias favors the answer in a particular slot.
- Verbosity bias favors the longer answer even when it adds nothing.
- Self-preference bias favors outputs from the judge’s own model family.
Mitigations match each one: swap positions, add “do not reward length” and a length-matched test case to your dataset, and use a judge from a different family than the system under test.
The most important mitigation is calibration against human labels. Label 30 to 50 outputs yourself, run the judge on the same set, and compare. Raw agreement misleads when most cases pass. If 90% of your cases are PASS, a judge that always says PASS agrees with you 90% of the time and tells you nothing. Cohen’s kappa corrects for chance agreement. With scikit-learn, you can compute it like so:
from sklearn.metrics import cohen_kappa_score
kappa = cohen_kappa_score(human_labels, judge_labels)
123
from sklearn.metrics import cohen_kappa_score kappa = cohen_kappa_score(human_labels, judge_labels)
A kappa of 0 means chance-level agreement, and 1 means perfect. Values above roughly 0.6 are usually read as substantial agreement. When kappa is low, read the disagreements, find what the judge misreads, and change the rubric. Anchoring each level of a scale with an example output also helps. Treat the labeled set as a regression test for the judge itself, and rerun it whenever you change the judge prompt or its model.
Evaluating RAG Systems #
A RAG (retrieval-augmented generation) system has two components that fail independently. The retriever finds context, and the generator writes an answer from it. Score them separately like so:
- Context precision: are the retrieved chunks relevant, with the best ones ranked first?
- Context recall: did retrieval find everything the answer needs?
- Faithfulness: is every claim in the answer supported by the retrieved context?
- Answer relevance: does the answer address the question that was asked?
But what happens when faithfulness is high and recall is low? The model stays grounded in what it received, and it received too little. Fix retrieval. High precision with low faithfulness points the other way: good documents, invented claims, so the fix lives in the prompt or the model.
Reference-free RAG metrics compute faithfulness and relevance without labeled answers, so they run on live traffic. Recall needs an expected answer, so it stays an offline metric. For hallucination checks with no context at all, sampling-based methods generate several answers to one prompt and compare them. Facts the model knows repeat across samples, while invented ones drift. The cost is several calls per answer.
Evaluating Agents and Conversations #
Agents add tool calls, planning, and loops. Start with task completion: did the user’s goal get done? A support agent with fluent replies and unresolved tickets fails at its job. When completion drops, three diagnostic scores locate the cause:
- Tool correctness: did the agent pick the right tool? This is usually an exact match against an expected tool list, with no judge involved.
- Argument correctness: were the parameters well formed and sensible for the input?
- Trajectory: did it take a reasonable path, without redundant calls or loops?
Multi-turn systems need the same treatment at the thread level. Each turn can look fine while the conversation forgets earlier context or contradicts itself, so give the judge the full history and score the outcome. Component scores sit under the end-to-end score: when completion falls, they show which step broke.
Running Evals Across the Release Cycle #
Offline evals run before you ship. Build a set of 25 to 50 cases drawn from representative queries, and cover normal requests, edge cases, adversarial inputs, and off-topic questions the bot should decline. Version the set with your code and refresh it as usage changes. Write release criteria in advance: minimum scores, a maximum regression against the current production version, and every safety check passing. Run the suite on each pull request and report which cases improved and which regressed.
Online evals run after you ship. Sample live traffic and score it with reference-free checks (faithfulness, tone, format) next to latency and cost. This catches inputs your dataset never imagined and quality drift when a provider updates a model.
Common challenges in LLM evaluation include overfitting to test sets, data leakage, and metric gaming. Use holdout sets, keep evaluation data separate, and track diverse metrics to assess quality. Choose metrics based on the use case and system architecture.
Summary & Further Reading #
The table summarizes what to do at each stage of an eval setup:
| Stage | What to do | Metric or technique |
|---|---|---|
| Scorer choice | Use code where a rule works, reference-based scorers for known answers, and a judge for subjective criteria. | Schema or regex checks, exact match, F1, LLM judge |
| Judge rubric | Define the criterion, inputs, steps, and a strict JSON output with reasoning before the verdict. | Temperature 0, majority vote over an odd number of runs |
| Judge scoring | Break the criterion into checkable claims, score a subjective one on a scale, or compare two outputs. | QAG (supported claims divided by total), G-Eval (probability-weighted score), pairwise with swapped order |
| Judge calibration | Label 30 to 50 outputs yourself and compare them with the judge. | Cohen’s kappa, position swaps, a judge from a different model family |
| RAG | Score the retriever and the generator separately. | Context precision, context recall, faithfulness, answer relevance |
| Agents | Check task completion first, then diagnose failures. | Tool correctness, argument correctness, trajectory |
| Offline evals | Run a versioned golden set on every pull request against written release criteria. | 25 to 50 cases, a holdout set, maximum allowed regression |
| Online evals | Sample live traffic and turn each production failure into a golden case. | Faithfulness, tone, format, latency, cost |
Here are a few resources for further learning:
- Define success criteria and build evaluations
- G-Eval: NLG evaluation using GPT-4 with better human alignment
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Ragas: automated evaluation of retrieval augmented generation
Happy learning!