# Your LLM Judge Needs a Test Suite > Source: > Published: 2026-07-08 11:01:19+00:00 Nobody ships a payment system without tests, but teams ship LLM judges into production on vibes every day. A grader, a triage classifier, an eval pipeline's scoring model — if an LLM's judgment gates something users care about, its behavior is a production dependency. And unlike code, it changes when you *didn't* change it: a model-version bump, a quiet provider update, a prompt tweak that fixed one case and broke five others you never noticed. The fix is boring and it works: regression tests. This is post 3 of the [assessment-first series](https://www.mpt.solutions/the-ai-tutor-everyone-builds-is-the-one-students-ignore/), and it ships [rubric-bench v0.1](https://github.com/michaeltuszynski/rubric-bench) — a small TypeScript library and CLI that treats the [grading engine from post 2](https://www.mpt.solutions/grading-written-answers-with-an-llm-properly/) like any other code under test. A **golden set** is a versioned JSON file of cases: a question (prompt, model answer, weighted rubric), one student answer, and the verdict a trustworthy grader must return: `correct` , `partial` , or `incorrect` . A **run** scores a grader against every case and reports accuracy overall and per tag. A **diff** compares two runs of the same set, case by case; a regression is a case that passed before and fails now. ``` rubric-bench run --set golden.json --out current.json --html report.html rubric-bench diff baseline.json current.json # exit 1 on regressions ``` The exit codes are the point. Wire `diff` into CI, and the build that changes your grading prompt fails when it breaks a case that used to pass. Grader behavior stops being a thing you discover from user complaints and becomes a thing your pipeline tells you before merge. The repo ships a 72-case golden set for introductory statistics: 60 genuine answers spread across correct, partial, and incorrect, written in actual undergraduate register (hedges, typos, half-remembered rules), plus 12 adversarial cases we'll get to next post. First full run of the post-2 grader on claude-sonnet-5: **69/72, 95.8%**. The three misses are the interesting part. All three were expected-`partial` , graded-`incorrect` — answers like "standard deviation is just easier to communicate than variance, so it became the convention." A human TA gives that half credit for knowing *that* SD is preferred while missing *why* (the units). The model judged the why-criterion unmet and the what-criterion unmet too, harshly but defensibly. None of the misses were expected-correct answers getting rejected, and none were wrong answers getting credit. The failure mode is concentrated exactly where human graders also disagree: the partial-credit gray zone. That's a distribution you can ship a pilot on, and precisely the region the tone-calibration work in post 5 targets. Two engineering notes from the first run, because reality bit immediately. One API response came back with no text block and killed the whole 72-case run; the runner now captures per-case errors as failed cases with an `error` field, and the reference grader retries once. And back-to-back identical runs differ by a case or two — [LLM-as-judge non-determinism is well documented](https://arxiv.org/abs/2306.05685) — so a single flipped case in a diff is noise, and a *pattern* of flips (three partials regressing after a prompt edit) is signal. The tool reports both; reading them is on you. The template transfers to any LLM-judgment task — support-ticket triage, content moderation, resume screening: `correct/partial/incorrect` . You can't meaningfully diff 7.2 vs 6.9.Sixty cases took one focused authoring session. That's the entire capital cost of knowing, forever, whether your judge still behaves. A grader that silently drifts mid-semester is a support catastrophe with a student body as witnesses; a triage model that drifts after a provider update mis-routes customers for weeks before anyone correlates the timing. The [Dartmouth pilot](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1s2.pdf) this series builds on ran its whole term on one pinned model for a reason. Pin the version, bench the swap, gate the prompt edits. It's the same discipline you already apply to schema migrations, applied to judgment. rubric-bench is regression testing, not psychometrics: it tells you the grader *changed*, not that it agrees with human raters at a publishable kappa; a real deployment eventually wants an inter-rater study against instructor grades, which this golden set can't substitute for. Seventy-two cases is a floor, not a benchmark; per-tag cells of 6 cases can't distinguish a real regression from noise at single-case granularity. And a golden set authored by one person inherits one person's judgment calls about partial credit — the honest fix is a second rater pass over the labels, which is on the roadmap. Next post: the 12 cases I skipped. Students will put prompt injections in their homework, and what the bench found when I attacked my own grader surprised me twice.