{"slug": "how-to-evaluate-an-llm-agent-evals-golden-sets-and-llm-as-judge", "title": "How to evaluate an LLM agent: evals, golden sets, and LLM-as-judge", "summary": "An engineer explains that LLM agents cannot be unit-tested for correctness due to their probabilistic nature, and instead require evals: scored, repeatable checks of acceptable outcomes. The approach combines golden sets grown from real failures, offline CI runs, online live-traffic checks, and LLM-as-judge scoring, with judges validated against human labels. The eval suite is wired into CI as a regression gate so quality drops fail the build.", "body_md": "**Short answer**\n\n**You can't unit-test an LLM to correctness, because the same input can take a different path on the next run.** Evals are the test suite for probabilistic systems: a scored, repeatable check of whether the system reached an acceptable outcome across runs, not whether it returned one exact string once.\n\nThe core toolkit is small: a curated golden set of input-to-expected-outcome examples, an offline run of that set in CI plus online checks against live traffic, and the right scorer for each task — assertion, golden set, LLM-as-judge, or human review. Wire it into CI as a regression gate and a quality drop fails the build.\n\nA unit test asserts that a deterministic function returns one exact value. An LLM is not deterministic: temperature, model updates, and the model's own sampling mean the same prompt can produce different — and differently worded — outputs across runs. Pinning to a single expected string makes a test that either flakes or asserts nothing useful.\n\nEvals replace that assertion with a different question: did the system reach an acceptable outcome, across multiple runs, at an acceptable rate? That reframes testing as a measurement problem. You score outputs against a target, aggregate over a set, and track the score over time. It is closer to an SLO with an error budget than to a pass/fail unit test, and it is the discipline that lets you change a prompt or swap a model without flying blind.\n\nA golden set (also called a reference or eval set) is a curated collection of input-to-expected-outcome examples. Each row pairs an input the system will actually see with the outcome you consider correct or acceptable — sometimes an exact answer, more often a rubric or a set of properties the answer must satisfy.\n\nThe most valuable golden sets are not invented up front; they are grown from real failures. Every production bug, every escaped edge case, every \"the model did something weird here\" becomes a new row. Over time the set encodes the actual shape of your traffic and your hardest cases, so a passing eval run means something concrete. Keep the set version-controlled, keep it representative rather than merely large, and treat adding a case after an incident as part of the fix.\n\nOffline evaluation runs your system against a fixed golden set, usually in CI, before anything ships. It is repeatable, comparable across changes, and cheap to run on every commit. Its limit is that it only measures the cases you thought to include.\n\nOnline evaluation measures behavior against live traffic and real outcomes after release — sampling production runs, scoring them, and watching real signals such as task completion, user corrections, or downstream success. It catches the distribution shift and the inputs your golden set never anticipated. Offline tells you a change is safe to ship; online tells you it actually worked. You need both, and online failures are the best source of new offline cases.\n\nFor subjective qualities — is this summary faithful, is this tone right, did the answer address the question — there is often no exact string to match against. LLM-as-judge uses a model to score the output against a rubric. It scales grading that would otherwise need a human on every row, and for many tasks it correlates well enough with human judgment to be useful.\n\nIt is also a component with real failure modes, and you should treat the judge as something you must itself evaluate:\n\nThe mitigation is to validate the judge against a sample of human-labeled data before you trust it, measure how well its scores agree with those labels, and re-check that agreement when you change the judge model or its rubric. A judge you have not validated is an unscored assumption, not a measurement.\n\nAn eval suite earns its keep when it runs automatically. Wire the offline golden-set run into CI so a change that regresses quality below a threshold fails the build, the same way a failing unit test or a dropped coverage number blocks a merge. That turns \"we think this prompt change is better\" into a measured claim, and it stops silent quality regressions from shipping when someone edits a prompt, upgrades a model, or refactors the harness.\n\nOne discipline matters more than the rest: pick the metric that reflects task success or outcome correctness, not just string similarity. Two answers can be worded completely differently and both be right; an exact-match score would fail a correct answer and a similarity score can pass a fluent wrong one. Score what you actually care about — did it do the job — and let the gate enforce that.\n\n| Eval approach | What it measures | Use when |\n|---|---|---|\n| Assertion / code checks | Hard, objective properties: valid JSON, required fields present, value in range, no banned content | The output has a checkable structure or invariant; cheapest and most reliable, so reach for it first |\n| Golden set | Outcome correctness against curated input-to-expected examples, aggregated across the set | You have known-good answers or rubrics and want a repeatable score you can gate CI on |\n| LLM-as-judge | Subjective quality — faithfulness, tone, relevance — scored by a model against a rubric | There is no exact answer to match and you need to scale grading; only after validating the judge against human labels |\n| Human review | Ground truth and nuanced judgment; the reference everything else is calibrated against | Stakes are high, the task is ambiguous, or you are validating a judge or building the golden set |\n\nEvals before scale\n\nBuild the eval harness before you scale usage, not after. Without it you have no way to tell whether a prompt change, a model upgrade, or a new tool made things better or worse — you are shipping on vibes. The team that can answer \"did quality go up or down, and by how much\" is the team that can iterate safely; the one that can't will eventually regress in production and not know why. Eval design is also the single strongest signal in hiring that a person has actually shipped with LLMs.\n\nThis page has just recommended wiring evals into CI and validating your judge against human labels. We do neither. Here is what actually gates a coach change on aiarch.dev, including both places it falls short of the advice above.\n\nTwo tiers. `npm run eval`\n\nscores the golden set against a mocked model — cheap, local, run on every coach change. Above it sits a live gate: real calls, scored for the behaviours that matter here (fading hints instead of revealing answers, naming the misconception, escalating after two stalled turns), and it has to clear 80% before a coach prompt or a deploy ships. It runs up to three times and takes the majority, exiting early once two runs pass. The coach is probabilistic even where the scoring isn't, so a verdict read off a single sample is noise.\n\n**There is no CI.** No `.github/workflows/`\n\nin the repo, and `git push`\n\ndeploys nothing — shipping is a deliberate `wrangler deploy`\n\n, with the gates as commands a human runs first. That is a gap, not a design choice. It is also why the offline tier has to stay fast: a gate you run beats a gate you describe.\n\nThe second gap is sharper. The live gate scores every reply against a rubric. **No model reads it.**\n\nThe scoring is deterministic assertions — eleven regexes over the coach's text, four that check which tools the turn actually called (did it fetch the lesson context, did it grade something the learner never attempted), two length floors, a `stop_reason`\n\ncheck. The harness is candid about the limits of that in its own comments: two of the checks are annotated as \"necessary-but-not-sufficient evidence\", naming a human or an LLM judge as the thing that *would* confirm real tutoring quality. So 80% measures observable adherence, against a threshold we chose rather than one calibrated against human labels. It catches regressions. It does not certify quality, and reading it as though it did is the mistake the section above warns about. See [the eval-gate pattern](https://aiarch.dev/patterns/eval-harness-gate) for how it slots into the release pipeline.\n\nThis is a conceptual overview; no specific benchmarks or figures are claimed, and API shapes change — verify against current provider docs before implementing. Corrections: [hello@aiarch.dev](mailto:hello@aiarch.dev).\n\n*Originally published at aiarch.dev/llm-evaluation-guide, where it is kept up to date.*\n\n*Want the skeleton instead of the essay? aiarch-templates has the src/lib/ seams, a threshold-gated eval stub and a cost-model skeleton. It is deliberately empty — it fixes the shape and you write the implementation. Apache-2.0.*", "url": "https://wpnews.pro/news/how-to-evaluate-an-llm-agent-evals-golden-sets-and-llm-as-judge", "canonical_source": "https://dev.to/aiarch_wibo/how-to-evaluate-an-llm-agent-evals-golden-sets-and-llm-as-judge-1e2g", "published_at": "2026-08-01 13:59:25+00:00", "updated_at": "2026-08-01 14:11:51.593462+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "mlops", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/how-to-evaluate-an-llm-agent-evals-golden-sets-and-llm-as-judge", "markdown": "https://wpnews.pro/news/how-to-evaluate-an-llm-agent-evals-golden-sets-and-llm-as-judge.md", "text": "https://wpnews.pro/news/how-to-evaluate-an-llm-agent-evals-golden-sets-and-llm-as-judge.txt", "jsonld": "https://wpnews.pro/news/how-to-evaluate-an-llm-agent-evals-golden-sets-and-llm-as-judge.jsonld"}}