# I got tired of breaking my LLM app with "one small prompt change", so I built a 450-line eval harness

> Source: <https://dev.to/litmustools/i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450-line-eval-4546>
> Published: 2026-11-11 09:45:27+00:00

Every LLM developer I know has the same workflow: tweak the prompt, eyeball three outputs, ship it. Then a user reports that the refund flow has been answering wrong for two days.

The fix is evals, but the popular options all asked for more than I wanted to give: a SaaS account, a framework dependency, a YAML religion. I wanted `pytest` energy: one file, run it, exit code.

So I wrote Litmus: a single Python file, stdlib only, that evals an LLM app against a JSONL dataset and gates regressions in CI.

```
{"id": "refund-q", "input": "How do I get a refund?",
 "expect": {"contains": ["30 days"], "regex": "refund"}}
```

That's it. Input, plus what a good answer must contain. Run it:

```
python3 litmus.py run --dataset cases.jsonl --endpoint https://api.openai.com/v1 --model gpt-4o-mini
LITMUS REPORT  ============================================
  pass rate   97.4%  (76/78)
  failed      1
  errored     1
  latency     mean 0.84s  p50 0.71s  p95 1.9s
  ============================================================
  [FAIL] refund-question (0.91s)
        contains: missing: 30 days
```

Run Litmus against your production prompt, save the JSON report as the baseline, and check every change against it:

```
python3 litmus.py compare baseline.json current.json --threshold 0.02
```

It lists the exact cases that used to pass and now fail, and exits 1 - so GitHub Actions blocks the merge. The workflow file is 15 lines, included in the guide.

`pip install` is where side tools go to die. Litmus is Python 3.8+ stdlib only. You can read all of it before running it.`contains` and `regex` are deterministic and free. LLM-as-judge is there for tone and helpfulness, with your rubric - and judge failures are reported as errors, never silently passed.
Litmus is $12, one time: [https://litmustools.gumroad.com/l/litmus](https://litmustools.gumroad.com/l/litmus). You get the tool, the field guide (eval-set patterns, LLM-judge rules, the CI workflow), and a runnable example. If it saves one bad deploy, it paid for itself.