{"slug": "your-llm-judge-needs-a-test-suite", "title": "Your LLM Judge Needs a Test Suite", "summary": "A developer released rubric-bench v0.1, a TypeScript library and CLI that treats LLM judges as code under test by running regression tests against a golden set of cases. The tool, which scored 95.8% on a 72-case introductory statistics set, catches regressions before deployment and addresses non-determinism in LLM-as-judge evaluations.", "body_md": "Nobody ships a payment system without tests, but teams ship LLM judges into production on vibes every day. A grader, a triage classifier, an eval pipeline's scoring model — if an LLM's judgment gates something users care about, its behavior is a production dependency. And unlike code, it changes when you *didn't* change it: a model-version bump, a quiet provider update, a prompt tweak that fixed one case and broke five others you never noticed.\n\nThe fix is boring and it works: regression tests. This is post 3 of the [assessment-first series](https://www.mpt.solutions/the-ai-tutor-everyone-builds-is-the-one-students-ignore/), and it ships [rubric-bench v0.1](https://github.com/michaeltuszynski/rubric-bench) — a small TypeScript library and CLI that treats the [grading engine from post 2](https://www.mpt.solutions/grading-written-answers-with-an-llm-properly/) like any other code under test.\n\nA **golden set** is a versioned JSON file of cases: a question (prompt, model answer, weighted rubric), one student answer, and the verdict a trustworthy grader must return: `correct`\n\n, `partial`\n\n, or `incorrect`\n\n. A **run** scores a grader against every case and reports accuracy overall and per tag. A **diff** compares two runs of the same set, case by case; a regression is a case that passed before and fails now.\n\n```\nrubric-bench run  --set golden.json --out current.json --html report.html\nrubric-bench diff baseline.json current.json     # exit 1 on regressions\n```\n\nThe exit codes are the point. Wire `diff`\n\ninto CI, and the build that changes your grading prompt fails when it breaks a case that used to pass. Grader behavior stops being a thing you discover from user complaints and becomes a thing your pipeline tells you before merge.\n\nThe repo ships a 72-case golden set for introductory statistics: 60 genuine answers spread across correct, partial, and incorrect, written in actual undergraduate register (hedges, typos, half-remembered rules), plus 12 adversarial cases we'll get to next post. First full run of the post-2 grader on claude-sonnet-5: **69/72, 95.8%**.\n\nThe three misses are the interesting part. All three were expected-`partial`\n\n, graded-`incorrect`\n\n— answers like \"standard deviation is just easier to communicate than variance, so it became the convention.\" A human TA gives that half credit for knowing *that* SD is preferred while missing *why* (the units). The model judged the why-criterion unmet and the what-criterion unmet too, harshly but defensibly. None of the misses were expected-correct answers getting rejected, and none were wrong answers getting credit. The failure mode is concentrated exactly where human graders also disagree: the partial-credit gray zone. That's a distribution you can ship a pilot on, and precisely the region the tone-calibration work in post 5 targets.\n\nTwo engineering notes from the first run, because reality bit immediately. One API response came back with no text block and killed the whole 72-case run; the runner now captures per-case errors as failed cases with an `error`\n\nfield, and the reference grader retries once. And back-to-back identical runs differ by a case or two — [LLM-as-judge non-determinism is well documented](https://arxiv.org/abs/2306.05685) — so a single flipped case in a diff is noise, and a *pattern* of flips (three partials regressing after a prompt edit) is signal. The tool reports both; reading them is on you.\n\nThe template transfers to any LLM-judgment task — support-ticket triage, content moderation, resume screening:\n\n`correct/partial/incorrect`\n\n. You can't meaningfully diff 7.2 vs 6.9.Sixty cases took one focused authoring session. That's the entire capital cost of knowing, forever, whether your judge still behaves.\n\nA grader that silently drifts mid-semester is a support catastrophe with a student body as witnesses; a triage model that drifts after a provider update mis-routes customers for weeks before anyone correlates the timing. The [Dartmouth pilot](https://intextbooks.science.uu.nl/workshop2026/files/itb26_s1s2.pdf) this series builds on ran its whole term on one pinned model for a reason. Pin the version, bench the swap, gate the prompt edits. It's the same discipline you already apply to schema migrations, applied to judgment.\n\nrubric-bench is regression testing, not psychometrics: it tells you the grader *changed*, not that it agrees with human raters at a publishable kappa; a real deployment eventually wants an inter-rater study against instructor grades, which this golden set can't substitute for. Seventy-two cases is a floor, not a benchmark; per-tag cells of 6 cases can't distinguish a real regression from noise at single-case granularity. And a golden set authored by one person inherits one person's judgment calls about partial credit — the honest fix is a second rater pass over the labels, which is on the roadmap.\n\nNext post: the 12 cases I skipped. Students will put prompt injections in their homework, and what the bench found when I attacked my own grader surprised me twice.", "url": "https://wpnews.pro/news/your-llm-judge-needs-a-test-suite", "canonical_source": "https://dev.to/michaeltuszynski/your-llm-judge-needs-a-test-suite-41oh", "published_at": "2026-07-08 11:01:19+00:00", "updated_at": "2026-07-08 11:28:49.181594+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-products", "ai-research", "mlops"], "entities": ["rubric-bench", "Claude Sonnet", "Dartmouth", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/your-llm-judge-needs-a-test-suite", "markdown": "https://wpnews.pro/news/your-llm-judge-needs-a-test-suite.md", "text": "https://wpnews.pro/news/your-llm-judge-needs-a-test-suite.txt", "jsonld": "https://wpnews.pro/news/your-llm-judge-needs-a-test-suite.jsonld"}}