{"slug": "i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450", "title": "I got tired of breaking my LLM app with \"one small prompt change\", so I built a 450-line eval harness", "summary": "An independent developer built Litmus, a single-file, stdlib-only Python eval harness that tests LLM applications against JSONL datasets and gates regressions in CI. The tool reports pass rates, failures and latency percentiles, supports a compare mode that exits non-zero when pass rates drop past a threshold, and uses deterministic contains/regex checks alongside an optional LLM-as-judge. It is sold for a one-time $12 with a field guide and CI workflow example.", "body_md": "Every LLM developer I know has the same workflow: tweak the prompt, eyeball three outputs, ship it. Then a user reports that the refund flow has been answering wrong for two days.\n\nThe fix is evals, but the popular options all asked for more than I wanted to give: a SaaS account, a framework dependency, a YAML religion. I wanted `pytest` energy: one file, run it, exit code.\n\nSo I wrote Litmus: a single Python file, stdlib only, that evals an LLM app against a JSONL dataset and gates regressions in CI.\n\n```\n{\"id\": \"refund-q\", \"input\": \"How do I get a refund?\",\n \"expect\": {\"contains\": [\"30 days\"], \"regex\": \"refund\"}}\n```\n\nThat's it. Input, plus what a good answer must contain. Run it:\n\n```\npython3 litmus.py run --dataset cases.jsonl --endpoint https://api.openai.com/v1 --model gpt-4o-mini\nLITMUS REPORT  ============================================\n  pass rate   97.4%  (76/78)\n  failed      1\n  errored     1\n  latency     mean 0.84s  p50 0.71s  p95 1.9s\n  ============================================================\n  [FAIL] refund-question (0.91s)\n        contains: missing: 30 days\n```\n\nRun Litmus against your production prompt, save the JSON report as the baseline, and check every change against it:\n\n```\npython3 litmus.py compare baseline.json current.json --threshold 0.02\n```\n\nIt lists the exact cases that used to pass and now fail, and exits 1 - so GitHub Actions blocks the merge. The workflow file is 15 lines, included in the guide.\n\n`pip install` is where side tools go to die. Litmus is Python 3.8+ stdlib only. You can read all of it before running it.`contains` and `regex` are deterministic and free. LLM-as-judge is there for tone and helpfulness, with your rubric - and judge failures are reported as errors, never silently passed.\nLitmus is $12, one time: [https://litmustools.gumroad.com/l/litmus](https://litmustools.gumroad.com/l/litmus). You get the tool, the field guide (eval-set patterns, LLM-judge rules, the CI workflow), and a runnable example. If it saves one bad deploy, it paid for itself.", "url": "https://wpnews.pro/news/i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450", "canonical_source": "https://dev.to/litmustools/i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450-line-eval-4546", "published_at": "2026-11-11 09:45:27+00:00", "updated_at": "2026-11-11 09:52:01.601405+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "mlops", "developer-tools"], "entities": ["Litmus", "GitHub Actions", "OpenAI", "gpt-4o-mini", "Gumroad"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450", "markdown": "https://wpnews.pro/news/i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450.md", "text": "https://wpnews.pro/news/i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450.txt", "jsonld": "https://wpnews.pro/news/i-got-tired-of-breaking-my-llm-app-with-one-small-prompt-change-so-i-built-a-450.jsonld"}}