Catch LLM regressions before your users do — a tiny CI gate for LLM output

You ship an LLM feature. Weeks later you tweak a prompt, or the provider rolls the model forward under you, and something breaks — not loudly, not in a stack trace, just three answers that used to be right and now aren't. Nobody notices until a user does.

I evaluate LLM output for a living, and this is the failure mode I see most. The fix isn't a platform or a dashboard. It's treating your eval like a test: a file you run on every change, that fails the build when it should. Here's how I do it, with runnable, dependency-free code.

Keep your evaluation data as plain JSON so anyone on the team can edit it without touching code. Each item is a question, the known-good answer, and — the important part — how to judge it:

[
  {"id": "q1", "question": "What year was the first moon landing?", "answer": "1969", "match": "contains"},
  {"id": "q2", "question": "What is pi to two decimals?", "answer": "3.14", "match": "numeric", "tol": 0.01},
  {"id": "q3", "question": "Capital of France?", "answer": "Paris", "match": "contains"}
]

Your model's answers are a simple {id: answer} object:

{
  "q1": "The first moon landing was in 1969.",
  "q2": "Pi is about 3.14159",
  "q3": "The capital of France is Paris."
}

Now score them. This is the open-source llm-eval-harness — one file, standard library only:

python llm_eval.py gold.json answers.json
accuracy: 100%  (3/3)

The match rule is what makes this trustworthy. contains passes "The first moon landing was in 1969." against gold 1969 — an exact-match-only script would mark that wrong and send you chasing a non-bug. numeric reads the first number in each and compares within a tolerance, so 3.14159 matches 3.14. The scorer normalizes first (lowercase, trim, collapse whitespace, drop trailing punctuation), which kills a whole class of false negatives.

And it exits non-zero the moment anything fails, so it drops straight into CI:

- name: LLM eval gate
  run: python llm_eval.py eval/gold.json eval/answers.json

That alone catches "did this change break a known-good answer?" on every PR.

Here's the trap: a change can raise your average score and still break the three questions your biggest customer depends on. The number that actually matters on a change isn't the score — it's which items that used to pass now fail.

So compare two runs and diff them. This is one of the pieces the LLM-Eval Starter Kit adds on top of the free harness:

from llm_eval_kit import run, diff_runs, load_gold, load_answers

gold   = load_gold("eval/gold.json")
before = run(gold, load_answers("eval/answers_main.json"), name="main")
after  = run(gold, load_answers("eval/answers_pr.json"),   name="pr")

diff = diff_runs(before, after)
print(f"accuracy {diff['before_accuracy']:.0%} -> {diff['after_accuracy']:.0%}")

if diff["regressed"]:
    print("REGRESSIONS:", ", ".join(diff["regressions"]))
    raise SystemExit(1)   # fail the build
php
accuracy 100% -> 67%
REGRESSIONS: q3, q5

diff_runs gives you the regressed ids (passed before, fail now), the fixed ids, and the score delta. Gate CI on regressed and a prompt change can't quietly ship a regression again — the PR goes red with the exact ids that broke.

String rules run out fast. "Is this summary faithful to the source?" "Did it follow the format?" "Is the tone right?" You can't contains-match those. The stable approach is an explicit rubric graded by a model at temperature 0 — not a vibe check.

The kit keeps that honest: the rubric is data (criteria + a numeric scale + a pass threshold), and the judge is any ask(prompt) -> str callable, so you bring your own model and key. There's also a deterministic offline judge so your test suite never needs an API key:

from llm_eval_kit import score, StubJudge, make_openai_judge

gold = [{
    "id": "sum1",
    "match": "rubric",
    "question": "Summarize the release note in one sentence.",
    "context": "v2.3 adds retry-with-backoff to the up and fixes a memory leak.",
    "rubric": {
        "criteria": [
            "Faithful to the source (no invented features)",
            "One sentence, plain language",
        ],
        "scale": [1, 5],
        "threshold": 4,
    },
}]
answers = {"sum1": "v2.3 adds automatic upload retries and fixes a memory leak."}

acc, failures = score(gold, answers, judge=StubJudge(must_include=["retry", "memory leak"]))

judge = make_openai_judge(model="gpt-4o-mini", temperature=0)  # reads OPENAI_API_KEY
acc, failures = score(gold, answers, judge=judge)

One rule I never skip: spot-check the judge against a handful of human labels. An LLM judge is a measuring instrument, and an uncalibrated instrument lies confidently. Grade ~20 answers yourself, compare, and only then trust it at scale.

For sampled or agentic systems you often care about "did any of k attempts pass?" rather than a single greedy answer. That's pass@k:

from llm_eval_kit import score_pass_at_k
rate, details = score_pass_at_k(gold, samples_by_id, k=5)   # samples_by_id: {id: [answer, ...]}

Same gold set, same match rules — you just feed it multiple samples per id.

You don't need a platform to stop shipping LLM regressions. You need:

The scoring core is open-source and free (MIT): llm-eval-harness. If you want the rubric LLM-as-judge, pass@k, regression diffs, and shareable HTML/Markdown/JSON reports ready to run — still zero dependencies — that's the LLM-Eval Starter Kit, launch price $24 for the first two weeks with code LAUNCH (then $39). There's also a short book on the whole method, Practical LLM Evaluation.

How are you catching regressions when a model updates under you? I'd genuinely like to hear it.