Catch LLM regressions before your users do — a tiny CI gate for LLM output

A developer has released an open-source tool, llm-eval-harness, designed to catch regressions in LLM output by integrating evaluation into CI pipelines. The tool uses plain JSON for test data and supports matching rules like 'contains' and 'numeric' to avoid false negatives. A companion kit, LLM-Eval Starter Kit, adds regression diffing and rubric-based grading with a model judge.

You ship an LLM feature. Weeks later you tweak a prompt, or the provider rolls the model forward under you, and something breaks — not loudly, not in a stack trace, just three answers that used to be right and now aren't. Nobody notices until a user does. I evaluate LLM output for a living, and this is the failure mode I see most. The fix isn't a platform or a dashboard. It's treating your eval like a test: a file you run on every change, that fails the build when it should. Here's how I do it, with runnable, dependency-free code. Keep your evaluation data as plain JSON so anyone on the team can edit it without touching code. Each item is a question, the known-good answer, and — the important part — how to judge it : {"id": "q1", "question": "What year was the first moon landing?", "answer": "1969", "match": "contains"}, {"id": "q2", "question": "What is pi to two decimals?", "answer": "3.14", "match": "numeric", "tol": 0.01}, {"id": "q3", "question": "Capital of France?", "answer": "Paris", "match": "contains"} Your model's answers are a simple {id: answer} object: { "q1": "The first moon landing was in 1969.", "q2": "Pi is about 3.14159", "q3": "The capital of France is Paris." } Now score them. This is the open-source llm-eval-harness https://github.com/zahid23saim/llm-eval-harness — one file, standard library only: python llm eval.py gold.json answers.json accuracy: 100% 3/3 The match rule is what makes this trustworthy. contains passes "The first moon landing was in 1969." against gold 1969 — an exact-match-only script would mark that wrong and send you chasing a non-bug. numeric reads the first number in each and compares within a tolerance, so 3.14159 matches 3.14 . The scorer normalizes first lowercase, trim, collapse whitespace, drop trailing punctuation , which kills a whole class of false negatives. And it exits non-zero the moment anything fails, so it drops straight into CI: .github/workflows/eval.yml - name: LLM eval gate run: python llm eval.py eval/gold.json eval/answers.json That alone catches "did this change break a known-good answer?" on every PR. Here's the trap: a change can raise your average score and still break the three questions your biggest customer depends on. The number that actually matters on a change isn't the score — it's which items that used to pass now fail. So compare two runs and diff them. This is one of the pieces the LLM-Eval Starter Kit https://saimzahid8.gumroad.com/l/llm-eval-kit adds on top of the free harness: python from llm eval kit import run, diff runs, load gold, load answers gold = load gold "eval/gold.json" before = run gold, load answers "eval/answers main.json" , name="main" after = run gold, load answers "eval/answers pr.json" , name="pr" diff = diff runs before, after print f"accuracy {diff 'before accuracy' :.0%} - {diff 'after accuracy' :.0%}" if diff "regressed" : print "REGRESSIONS:", ", ".join diff "regressions" raise SystemExit 1 fail the build php accuracy 100% - 67% REGRESSIONS: q3, q5 diff runs gives you the regressed ids passed before, fail now , the fixed ids, and the score delta. Gate CI on regressed and a prompt change can't quietly ship a regression again — the PR goes red with the exact ids that broke. String rules run out fast. "Is this summary faithful to the source?" "Did it follow the format?" "Is the tone right?" You can't contains -match those. The stable approach is an explicit rubric graded by a model at temperature 0 — not a vibe check. The kit keeps that honest: the rubric is data criteria + a numeric scale + a pass threshold , and the judge is any ask prompt - str callable, so you bring your own model and key. There's also a deterministic offline judge so your test suite never needs an API key : python from llm eval kit import score, StubJudge, make openai judge gold = { "id": "sum1", "match": "rubric", "question": "Summarize the release note in one sentence.", "context": "v2.3 adds retry-with-backoff to the uploader and fixes a memory leak.", "rubric": { "criteria": "Faithful to the source no invented features ", "One sentence, plain language", , "scale": 1, 5 , "threshold": 4, }, } answers = {"sum1": "v2.3 adds automatic upload retries and fixes a memory leak."} Offline + deterministic — perfect for CI/tests, no key: acc, failures = score gold, answers, judge=StubJudge must include= "retry", "memory leak" Real judge — bring your own key OpenAI / OpenRouter / a local server : judge = make openai judge model="gpt-4o-mini", temperature=0 reads OPENAI API KEY acc, failures = score gold, answers, judge=judge One rule I never skip: spot-check the judge against a handful of human labels. An LLM judge is a measuring instrument, and an uncalibrated instrument lies confidently. Grade ~20 answers yourself, compare, and only then trust it at scale. For sampled or agentic systems you often care about "did any of k attempts pass?" rather than a single greedy answer. That's pass@k: python from llm eval kit import score pass at k rate, details = score pass at k gold, samples by id, k=5 samples by id: {id: answer, ... } Same gold set, same match rules — you just feed it multiple samples per id. You don't need a platform to stop shipping LLM regressions. You need: The scoring core is open-source and free MIT : llm-eval-harness https://github.com/zahid23saim/llm-eval-harness . If you want the rubric LLM-as-judge, pass@k, regression diffs, and shareable HTML/Markdown/JSON reports ready to run — still zero dependencies — that's the LLM-Eval Starter Kit https://saimzahid8.gumroad.com/l/llm-eval-kit , launch price $24 for the first two weeks with code LAUNCH then $39 . There's also a short book on the whole method, Practical LLM Evaluation https://saimzahid8.gumroad.com/l/practical-llm-evaluation . How are you catching regressions when a model updates under you? I'd genuinely like to hear it.