Nobody ships a payment system without tests, but teams ship LLM judges into production on vibes every day. A grader, a triage classifier, an eval pipeline's scoring model — if an LLM's judgment gates something users care about, its behavior is a production dependency. And unlike code, it changes when you didn't change it: a model-version bump, a quiet provider update, a prompt tweak that fixed one case and broke five others you never noticed.
The fix is boring and it works: regression tests. This is post 3 of the assessment-first series, and it ships rubric-bench v0.1 — a small TypeScript library and CLI that treats the grading engine from post 2 like any other code under test.
A golden set is a versioned JSON file of cases: a question (prompt, model answer, weighted rubric), one student answer, and the verdict a trustworthy grader must return: correct
, partial
, or incorrect
. A run scores a grader against every case and reports accuracy overall and per tag. A diff compares two runs of the same set, case by case; a regression is a case that passed before and fails now.
rubric-bench run --set golden.json --out current.json --html report.html
rubric-bench diff baseline.json current.json # exit 1 on regressions
The exit codes are the point. Wire diff
into CI, and the build that changes your grading prompt fails when it breaks a case that used to pass. Grader behavior stops being a thing you discover from user complaints and becomes a thing your pipeline tells you before merge.
The repo ships a 72-case golden set for introductory statistics: 60 genuine answers spread across correct, partial, and incorrect, written in actual undergraduate register (hedges, typos, half-remembered rules), plus 12 adversarial cases we'll get to next post. First full run of the post-2 grader on claude-sonnet-5: 69/72, 95.8%.
The three misses are the interesting part. All three were expected-partial
, graded-incorrect
— answers like "standard deviation is just easier to communicate than variance, so it became the convention." A human TA gives that half credit for knowing that SD is preferred while missing why (the units). The model judged the why-criterion unmet and the what-criterion unmet too, harshly but defensibly. None of the misses were expected-correct answers getting rejected, and none were wrong answers getting credit. The failure mode is concentrated exactly where human graders also disagree: the partial-credit gray zone. That's a distribution you can ship a pilot on, and precisely the region the tone-calibration work in post 5 targets.
Two engineering notes from the first run, because reality bit immediately. One API response came back with no text block and killed the whole 72-case run; the runner now captures per-case errors as failed cases with an error
field, and the reference grader retries once. And back-to-back identical runs differ by a case or two — LLM-as-judge non-determinism is well documented — so a single flipped case in a diff is noise, and a pattern of flips (three partials regressing after a prompt edit) is signal. The tool reports both; reading them is on you.
The template transfers to any LLM-judgment task — support-ticket triage, content moderation, resume screening:
correct/partial/incorrect
. You can't meaningfully diff 7.2 vs 6.9.Sixty cases took one focused authoring session. That's the entire capital cost of knowing, forever, whether your judge still behaves.
A grader that silently drifts mid-semester is a support catastrophe with a student body as witnesses; a triage model that drifts after a provider update mis-routes customers for weeks before anyone correlates the timing. The Dartmouth pilot this series builds on ran its whole term on one pinned model for a reason. Pin the version, bench the swap, gate the prompt edits. It's the same discipline you already apply to schema migrations, applied to judgment.
rubric-bench is regression testing, not psychometrics: it tells you the grader changed, not that it agrees with human raters at a publishable kappa; a real deployment eventually wants an inter-rater study against instructor grades, which this golden set can't substitute for. Seventy-two cases is a floor, not a benchmark; per-tag cells of 6 cases can't distinguish a real regression from noise at single-case granularity. And a golden set authored by one person inherits one person's judgment calls about partial credit — the honest fix is a second rater pass over the labels, which is on the roadmap.
Next post: the 12 cases I skipped. Students will put prompt injections in their homework, and what the bench found when I attacked my own grader surprised me twice.