I got tired of breaking my LLM app with "one small prompt change", so I built a 450-line eval harness

An independent developer built Litmus, a single-file, stdlib-only Python eval harness that tests LLM applications against JSONL datasets and gates regressions in CI. The tool reports pass rates, failures and latency percentiles, supports a compare mode that exits non-zero when pass rates drop past a threshold, and uses deterministic contains/regex checks alongside an optional LLM-as-judge. It is sold for a one-time $12 with a field guide and CI workflow example.

Every LLM developer I know has the same workflow: tweak the prompt, eyeball three outputs, ship it. Then a user reports that the refund flow has been answering wrong for two days. The fix is evals, but the popular options all asked for more than I wanted to give: a SaaS account, a framework dependency, a YAML religion. I wanted pytest energy: one file, run it, exit code. So I wrote Litmus: a single Python file, stdlib only, that evals an LLM app against a JSONL dataset and gates regressions in CI. {"id": "refund-q", "input": "How do I get a refund?", "expect": {"contains": "30 days" , "regex": "refund"}} That's it. Input, plus what a good answer must contain. Run it: python3 litmus.py run --dataset cases.jsonl --endpoint https://api.openai.com/v1 --model gpt-4o-mini LITMUS REPORT ============================================ pass rate 97.4% 76/78 failed 1 errored 1 latency mean 0.84s p50 0.71s p95 1.9s ============================================================ FAIL refund-question 0.91s contains: missing: 30 days Run Litmus against your production prompt, save the JSON report as the baseline, and check every change against it: python3 litmus.py compare baseline.json current.json --threshold 0.02 It lists the exact cases that used to pass and now fail, and exits 1 - so GitHub Actions blocks the merge. The workflow file is 15 lines, included in the guide. pip install is where side tools go to die. Litmus is Python 3.8+ stdlib only. You can read all of it before running it. contains and regex are deterministic and free. LLM-as-judge is there for tone and helpfulness, with your rubric - and judge failures are reported as errors, never silently passed. Litmus is $12, one time: https://litmustools.gumroad.com/l/litmus https://litmustools.gumroad.com/l/litmus . You get the tool, the field guide eval-set patterns, LLM-judge rules, the CI workflow , and a runnable example. If it saves one bad deploy, it paid for itself.