{"slug": "catch-prompt-regressions-before-they-ship-testing-an-invoice-extraction-prompt", "title": "Catch Prompt Regressions Before They Ship: Testing an Invoice-Extraction Prompt with Promptfoo and GitHub Actions", "summary": "A developer published a hands-on guide and example repo showing how to regression-test an LLM invoice-extraction prompt with Promptfoo and GitHub Actions, running two prompt versions side by side against 20 generated invoices on every pull request. The setup stacks four assertions per test case — JSON schema validation, exact field checks with float tolerance, internal arithmetic consistency, and edge-case coverage — and fails the build when extraction quality drops, catching failures like euro invoices being booked in dollars.", "body_md": "*This article was written with help from an AI assistant; all code was tested and the text reviewed. Example repo: [https://github.com/zackthompson317/promptfoo-invoice-extraction-ci](https://github.com/zackthompson317/promptfoo-invoice-extraction-ci) (Promptfoo 0.125.0, Node 22.22+)*\n\nSay your accounts-payable tool sends every incoming invoice to an LLM and asks for JSON: invoice number, vendor, date, currency, line items, subtotal, tax and total. It works. Then someone \"improves\" the prompt so it handles a new vendor's layout, and three weeks later finance notices that every euro invoice has been booked in dollars.\n\nNothing crashed. The output was still valid JSON, and nobody's unit tests touched the prompt, because prompts usually live outside the test suite. That's the gap this tutorial closes. We'll put a prompt under the same kind of regression testing as code, using Promptfoo's declarative test cases, and run it on every pull request with GitHub Actions. The build fails if quality drops.\n\nBy the end you'll have:\n\n`promptfooconfig.yaml` that compares two prompt versions side by side\n\n```\nprompts/v1.txt              # the original, terse prompt\nprompts/v2.txt              # the \"improved\" prompt under review\ninvoices/inv01.txt..inv20.txt\ntests.yaml                  # one test case per invoice\nschema.json                 # JSON schema for the extraction output\nasserts/totals-add-up.js    # custom arithmetic check\nproviders/local-extractor.js# offline stand-in model for $0 CI\npromptfooconfig.yaml\n.github/workflows/prompt-eval.yml\n```\n\nVersion 1 is what most teams write first:\n\n```\nExtract the invoice below as JSON with keys invoice_number, vendor, date (YYYY-MM-DD),\ncurrency (ISO code), line_items [{description, quantity, unit_price}], subtotal, tax, total.\nReturn JSON only.\n\n{{invoice}}\n```\n\nVersion 2 adds the constraints we learned the hard way: no code fences, currency inferred from the symbol, plain numbers, and tolerance for a `TOTAL` label as well as `Total Due`.\n\nPromptfoo treats `{{invoice}}` as a Nunjucks variable, and each test case fills it in. Listing both files under `prompts:` makes every test run against both versions, so the results grid becomes a before/after comparison.\n\nReal invoices contain names, addresses and bank details, so they shouldn't go into a public repo or a CI log. The example repo uses 20 generated invoices with fake vendors and random line items. Tax is 7%, and the totals are computed, so they're guaranteed to be correct. Two invoices are in euros, and two are deliberate edge cases:\n\n`inv19` labels the total `TOTAL` instead of `Total Due`\n`inv20` starts with a chatty note before the invoice header\nEdge cases are where prompt changes usually break, so give them their own tests rather than hoping the happy path covers them. As you find production failures, add a sanitized copy of each one to `invoices/` and it becomes a permanent regression test.\n\nA single \"looks right\" check isn't enough. The suite stacks four assertions on each test, from cheap and general to strict and specific.\n\n**1. Valid JSON matching a schema.** Promptfoo's `is-json` assertion takes an optional JSON schema:\n\n```\n- type: is-json\n  value: file://schema.json\n```\n\nThe schema requires every field, restricts `currency` to `USD` or `EUR`, requires a `YYYY-MM-DD` date, and needs at least one line item. That catches code fences, prose before the JSON, missing keys, and `\"$1,234.00\"` strings where numbers belong.\n\n**2. Exact field checks.** For each invoice we know the right answer, so we assert it:\n\n``` js\n- type: javascript\n  value: |\n    const o = JSON.parse(output);\n    return o.invoice_number === 'INV-1007' && o.currency === 'EUR' && Math.abs(o.total - 1043.29) < 0.01;\n```\n\nThis is the assertion that would have caught the euro bug in the opening story. Compare floats with a tolerance, never `===`.\n\n**3. Internal consistency.** A model can copy the total correctly and still invent a line item. `asserts/totals-add-up.js` recomputes the math:\n\n``` js\nmodule.exports = (output) => {\n  const o = JSON.parse(output);\n  const sum = o.line_items.reduce((s, l) => s + l.quantity * l.unit_price, 0);\n  const ok = Math.abs(sum - o.subtotal) < 0.01 && Math.abs(o.subtotal + o.tax - o.total) < 0.01;\n  return { pass: ok, score: ok ? 1 : 0,\n           reason: ok ? 'totals reconcile' : `sum ${sum.toFixed(2)} vs subtotal ${o.subtotal}` };\n};\n```\n\nReturning `{pass, score, reason}` instead of a bare boolean means the Promptfoo viewer shows *why* a case failed, which saves a lot of time when you're debugging twenty red cells.\n\n**4. Rubric grading for fuzzy fields (optional).** Some fields don't have one right string. \"Acme Supplies\" and \"Acme Supplies Inc.\" are both acceptable vendor names. Promptfoo's `llm-rubric` assertion asks a grading model to judge:\n\n```\n- type: llm-rubric\n  value: The vendor field names the same company as the \"From:\" line, ignoring suffixes like Inc or LLC.\n```\n\n`llm-rubric` needs a real grading model and an API key, so the example repo leaves it off by default. Turn it on once you add a provider key (see Step 6).\n\nHere's a practical problem: you want the pipeline in place on day one, but you may not want to put an API key in CI yet, or pay for evals on every documentation PR. Promptfoo supports custom providers, which are plain JavaScript classes with a `callApi` method, so the repo ships a deterministic stand-in:\n\n```\nclass LocalExtractor {\n  id() { return 'local-extractor'; }\n  async callApi(prompt) {\n    const t = prompt.slice(prompt.indexOf('INVOICE'));\n    // ...regex extraction of number, vendor, date, line items, totals...\n    return { output: JSON.stringify(out) };\n  }\n}\nmodule.exports = LocalExtractor;\n```\n\nIt's wired in with `id: file://providers/local-extractor.js`. To be clear, this doesn't measure model quality. It parses the invoice text out of the rendered prompt. What it does show is that your config, tests, schema, custom assertions and CI gate all work, for free, before any model is involved. It's also a useful baseline: if a real model scores below a regex parser on clean synthetic data, you'll want to know.\n\nRunning it locally:\n\n```\nnpx promptfoo@0.125.0 eval -c promptfooconfig.yaml --no-cache\nnpx promptfoo@0.125.0 view\n```\n\nThe output is a grid with one row per invoice and one column per prompt. With the local provider, all 40 cells (20 tests × 2 prompts) pass.\n\nThe workflow runs whenever a PR touches prompts, tests, invoices, assertions or providers:\n\n```\non:\n  pull_request:\n    paths: ['prompts/**', 'promptfooconfig.yaml', 'tests.yaml', 'invoices/**', 'asserts/**', 'providers/**']\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-node@v4\n        with: { node-version: 22.22 }\n      - run: npx --yes promptfoo@0.125.0 eval -c promptfooconfig.yaml --no-cache -o results.json\n        env: { PROMPTFOO_DISABLE_TELEMETRY: 1 }\n      - name: Fail if pass rate < 95%\n        run: |\n          node -e \"const r=require('./results.json').results.stats;\n          const rate=r.successes/(r.successes+r.failures);\n          console.log('pass rate',rate); if(rate<0.95) process.exit(1)\"\n      - uses: actions/upload-artifact@v4\n        if: always()\n        with: { name: promptfoo-results, path: results.json }\n```\n\nA few design choices are worth calling out:\n\n`npx promptfoo` will eventually change under you. Pin the model version too, once you add one.`temperature: 0` for extraction tasks.`results.json` even on failure means reviewers can download the exact outputs that failed without re-running anything.`--no-cache`), so a passing cached result can't hide a regression after a provider-side model update.\nWhen you're ready, uncomment a provider in `promptfooconfig.yaml`:\n\n```\nproviders:\n  - id: file://providers/local-extractor.js\n    label: local-extractor\n  - id: openai:gpt-4.1-mini\n    config: { temperature: 0 }\n```\n\nAdd the key as a repository secret (`OPENAI_API_KEY`) and pass it through the workflow's `env`. Keeping the local extractor alongside the real model gives you a stable baseline column in every run. Now `llm-rubric` can be turned on too.\n\nOn cost: 20 short invoices × 2 prompts × 1 model is 40 small requests per run, so with a mini-sized model a run costs well under a cent. If that adds up across many PRs, the `paths:` filter already skips runs for changes that can't affect prompt behavior.\n\n`promptfoo view` locally, and filter to failures.`reason` string from the custom assertion. \"sum 812.40 vs subtotal 912.40\" points straight at a dropped line item.\nClone the repo, run `npx promptfoo@0.125.0 eval`, then try breaking `prompts/v2.txt`, for example by deleting the currency instruction, and watch the gate catch it.", "url": "https://wpnews.pro/news/catch-prompt-regressions-before-they-ship-testing-an-invoice-extraction-prompt", "canonical_source": "https://dev.to/zackthompson317/catch-prompt-regressions-before-they-ship-testing-an-invoice-extraction-prompt-with-promptfoo-and-3me9", "published_at": "2026-10-25 04:06:13+00:00", "updated_at": "2026-10-25 04:11:12.579989+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools", "mlops", "ai-agents"], "entities": ["Promptfoo", "GitHub Actions", "GitHub", "Node.js"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/catch-prompt-regressions-before-they-ship-testing-an-invoice-extraction-prompt", "markdown": "https://wpnews.pro/news/catch-prompt-regressions-before-they-ship-testing-an-invoice-extraction-prompt.md", "text": "https://wpnews.pro/news/catch-prompt-regressions-before-they-ship-testing-an-invoice-extraction-prompt.txt", "jsonld": "https://wpnews.pro/news/catch-prompt-regressions-before-they-ship-testing-an-invoice-extraction-prompt.jsonld"}}