{"slug": "catch-llm-regressions-before-your-users-do-a-tiny-ci-gate-for-llm-output", "title": "Catch LLM regressions before your users do — a tiny CI gate for LLM output", "summary": "A developer has released an open-source tool, llm-eval-harness, designed to catch regressions in LLM output by integrating evaluation into CI pipelines. The tool uses plain JSON for test data and supports matching rules like 'contains' and 'numeric' to avoid false negatives. A companion kit, LLM-Eval Starter Kit, adds regression diffing and rubric-based grading with a model judge.", "body_md": "You ship an LLM feature. Weeks later you tweak a prompt, or the provider rolls the model forward under you, and something breaks — not loudly, not in a stack trace, just three answers that used to be right and now aren't. Nobody notices until a user does.\n\nI evaluate LLM output for a living, and this is the failure mode I see most. The fix isn't a platform or a dashboard. It's treating your eval like a test: **a file you run on every change, that fails the build when it should.** Here's how I do it, with runnable, dependency-free code.\n\nKeep your evaluation data as plain JSON so anyone on the team can edit it without touching code. Each item is a question, the known-good answer, and — the important part — *how to judge it*:\n\n```\n[\n  {\"id\": \"q1\", \"question\": \"What year was the first moon landing?\", \"answer\": \"1969\", \"match\": \"contains\"},\n  {\"id\": \"q2\", \"question\": \"What is pi to two decimals?\", \"answer\": \"3.14\", \"match\": \"numeric\", \"tol\": 0.01},\n  {\"id\": \"q3\", \"question\": \"Capital of France?\", \"answer\": \"Paris\", \"match\": \"contains\"}\n]\n```\n\nYour model's answers are a simple `{id: answer}` object:\n\n```\n{\n  \"q1\": \"The first moon landing was in 1969.\",\n  \"q2\": \"Pi is about 3.14159\",\n  \"q3\": \"The capital of France is Paris.\"\n}\n```\n\nNow score them. This is the open-source [`llm-eval-harness`](https://github.com/zahid23saim/llm-eval-harness) — one file, standard library only:\n\n```\npython llm_eval.py gold.json answers.json\naccuracy: 100%  (3/3)\n```\n\nThe `match` rule is what makes this trustworthy. `contains` passes *\"The first moon landing was in 1969.\"* against gold `1969` — an exact-match-only script would mark that wrong and send you chasing a non-bug. `numeric` reads the first number in each and compares within a tolerance, so `3.14159` matches `3.14`. The scorer normalizes first (lowercase, trim, collapse whitespace, drop trailing punctuation), which kills a whole class of false negatives.\n\nAnd it exits non-zero the moment anything fails, so it drops straight into CI:\n\n```\n# .github/workflows/eval.yml\n- name: LLM eval gate\n  run: python llm_eval.py eval/gold.json eval/answers.json\n```\n\nThat alone catches \"did this change break a known-good answer?\" on every PR.\n\nHere's the trap: a change can *raise* your average score and still break the three questions your biggest customer depends on. The number that actually matters on a change isn't the score — it's **which items that used to pass now fail.**\n\nSo compare two runs and diff them. This is one of the pieces the [LLM-Eval Starter Kit](https://saimzahid8.gumroad.com/l/llm-eval-kit) adds on top of the free harness:\n\n``` python\nfrom llm_eval_kit import run, diff_runs, load_gold, load_answers\n\ngold   = load_gold(\"eval/gold.json\")\nbefore = run(gold, load_answers(\"eval/answers_main.json\"), name=\"main\")\nafter  = run(gold, load_answers(\"eval/answers_pr.json\"),   name=\"pr\")\n\ndiff = diff_runs(before, after)\nprint(f\"accuracy {diff['before_accuracy']:.0%} -> {diff['after_accuracy']:.0%}\")\n\nif diff[\"regressed\"]:\n    print(\"REGRESSIONS:\", \", \".join(diff[\"regressions\"]))\n    raise SystemExit(1)   # fail the build\nphp\naccuracy 100% -> 67%\nREGRESSIONS: q3, q5\n```\n\n`diff_runs` gives you the regressed ids (passed before, fail now), the fixed ids, and the score delta. Gate CI on `regressed` and a prompt change can't quietly ship a regression again — the PR goes red with the exact ids that broke.\n\nString rules run out fast. \"Is this summary faithful to the source?\" \"Did it follow the format?\" \"Is the tone right?\" You can't `contains`-match those. The stable approach is an **explicit rubric graded by a model at temperature 0** — not a vibe check.\n\nThe kit keeps that honest: the rubric is data (criteria + a numeric scale + a pass threshold), and the judge is any `ask(prompt) -> str` callable, so you bring your own model and key. There's also a deterministic offline judge so your **test suite never needs an API key**:\n\n``` python\nfrom llm_eval_kit import score, StubJudge, make_openai_judge\n\ngold = [{\n    \"id\": \"sum1\",\n    \"match\": \"rubric\",\n    \"question\": \"Summarize the release note in one sentence.\",\n    \"context\": \"v2.3 adds retry-with-backoff to the uploader and fixes a memory leak.\",\n    \"rubric\": {\n        \"criteria\": [\n            \"Faithful to the source (no invented features)\",\n            \"One sentence, plain language\",\n        ],\n        \"scale\": [1, 5],\n        \"threshold\": 4,\n    },\n}]\nanswers = {\"sum1\": \"v2.3 adds automatic upload retries and fixes a memory leak.\"}\n\n# Offline + deterministic — perfect for CI/tests, no key:\nacc, failures = score(gold, answers, judge=StubJudge(must_include=[\"retry\", \"memory leak\"]))\n\n# Real judge — bring your own key (OpenAI / OpenRouter / a local server):\njudge = make_openai_judge(model=\"gpt-4o-mini\", temperature=0)  # reads OPENAI_API_KEY\nacc, failures = score(gold, answers, judge=judge)\n```\n\nOne rule I never skip: **spot-check the judge against a handful of human labels.** An LLM judge is a measuring instrument, and an uncalibrated instrument lies confidently. Grade ~20 answers yourself, compare, and only then trust it at scale.\n\nFor sampled or agentic systems you often care about \"did *any* of k attempts pass?\" rather than a single greedy answer. That's pass@k:\n\n``` python\nfrom llm_eval_kit import score_pass_at_k\nrate, details = score_pass_at_k(gold, samples_by_id, k=5)   # samples_by_id: {id: [answer, ...]}\n```\n\nSame gold set, same match rules — you just feed it multiple samples per id.\n\nYou don't need a platform to stop shipping LLM regressions. You need:\n\nThe scoring core is open-source and free (MIT): **[llm-eval-harness](https://github.com/zahid23saim/llm-eval-harness)**. If you want the rubric LLM-as-judge, pass@k, regression diffs, and shareable HTML/Markdown/JSON reports ready to run — still zero dependencies — that's the **[LLM-Eval Starter Kit](https://saimzahid8.gumroad.com/l/llm-eval-kit)**, launch price **$24** for the first two weeks with code `LAUNCH` (then $39). There's also a short book on the whole method, **[Practical LLM Evaluation](https://saimzahid8.gumroad.com/l/practical-llm-evaluation)**.\n\nHow are you catching regressions when a model updates under you? I'd genuinely like to hear it.", "url": "https://wpnews.pro/news/catch-llm-regressions-before-your-users-do-a-tiny-ci-gate-for-llm-output", "canonical_source": "https://dev.to/zahid23saim/catch-llm-regressions-before-your-users-do-a-tiny-ci-gate-for-llm-output-8bg", "published_at": "2026-09-07 13:38:57+00:00", "updated_at": "2026-09-07 13:57:08.514145+00:00", "lang": "en", "topics": ["developer-tools", "mlops", "large-language-models"], "entities": ["llm-eval-harness", "LLM-Eval Starter Kit", "GitHub", "Saim Zahid"], "alternates": {"html": "https://wpnews.pro/news/catch-llm-regressions-before-your-users-do-a-tiny-ci-gate-for-llm-output", "markdown": "https://wpnews.pro/news/catch-llm-regressions-before-your-users-do-a-tiny-ci-gate-for-llm-output.md", "text": "https://wpnews.pro/news/catch-llm-regressions-before-your-users-do-a-tiny-ci-gate-for-llm-output.txt", "jsonld": "https://wpnews.pro/news/catch-llm-regressions-before-your-users-do-a-tiny-ci-gate-for-llm-output.jsonld"}}