Every LLM developer I know has the same workflow: tweak the prompt, eyeball three outputs, ship it. Then a user reports that the refund flow has been answering wrong for two days.
The fix is evals, but the popular options all asked for more than I wanted to give: a SaaS account, a framework dependency, a YAML religion. I wanted pytest energy: one file, run it, exit code.
So I wrote Litmus: a single Python file, stdlib only, that evals an LLM app against a JSONL dataset and gates regressions in CI.
{"id": "refund-q", "input": "How do I get a refund?",
"expect": {"contains": ["30 days"], "regex": "refund"}}
That's it. Input, plus what a good answer must contain. Run it:
python3 litmus.py run --dataset cases.jsonl --endpoint https://api.openai.com/v1 --model gpt-4o-mini
LITMUS REPORT ============================================
pass rate 97.4% (76/78)
failed 1
errored 1
[FAIL] refund-question (0.91s)
contains: missing: 30 days
Run Litmus against your production prompt, save the JSON report as the baseline, and check every change against it:
python3 litmus.py compare baseline.json current.json --threshold 0.02
It lists the exact cases that used to pass and now fail, and exits 1 - so GitHub Actions blocks the merge. The workflow file is 15 lines, included in the guide.
pip install is where side tools go to die. Litmus is Python 3.8+ stdlib only. You can read all of it before running it.contains and regex are deterministic and free. LLM-as-judge is there for tone and helpfulness, with your rubric - and judge failures are reported as errors, never silently passed.
Litmus is $12, one time: https://litmustools.gumroad.com/l/litmus. You get the tool, the field guide (eval-set patterns, LLM-judge rules, the CI workflow), and a runnable example. If it saves one bad deploy, it paid for itself.