This article was written with help from an AI assistant; all code was tested and the text reviewed. Example repo: https://github.com/zackthompson317/promptfoo-invoice-extraction-ci (Promptfoo 0.125.0, Node 22.22+)
Say your accounts-payable tool sends every incoming invoice to an LLM and asks for JSON: invoice number, vendor, date, currency, line items, subtotal, tax and total. It works. Then someone "improves" the prompt so it handles a new vendor's layout, and three weeks later finance notices that every euro invoice has been booked in dollars.
Nothing crashed. The output was still valid JSON, and nobody's unit tests touched the prompt, because prompts usually live outside the test suite. That's the gap this tutorial closes. We'll put a prompt under the same kind of regression testing as code, using Promptfoo's declarative test cases, and run it on every pull request with GitHub Actions. The build fails if quality drops.
By the end you'll have:
promptfooconfig.yaml that compares two prompt versions side by side
prompts/v1.txt # the original, terse prompt
prompts/v2.txt # the "improved" prompt under review
invoices/inv01.txt..inv20.txt
tests.yaml # one test case per invoice
schema.json # JSON schema for the extraction output
asserts/totals-add-up.js # custom arithmetic check
providers/local-extractor.js# offline stand-in model for $0 CI
promptfooconfig.yaml
.github/workflows/prompt-eval.yml
Version 1 is what most teams write first:
Extract the invoice below as JSON with keys invoice_number, vendor, date (YYYY-MM-DD),
currency (ISO code), line_items [{description, quantity, unit_price}], subtotal, tax, total.
Return JSON only.
{{invoice}}
Version 2 adds the constraints we learned the hard way: no code fences, currency inferred from the symbol, plain numbers, and tolerance for a TOTAL label as well as Total Due.
Promptfoo treats {{invoice}} as a Nunjucks variable, and each test case fills it in. Listing both files under prompts: makes every test run against both versions, so the results grid becomes a before/after comparison.
Real invoices contain names, addresses and bank details, so they shouldn't go into a public repo or a CI log. The example repo uses 20 generated invoices with fake vendors and random line items. Tax is 7%, and the totals are computed, so they're guaranteed to be correct. Two invoices are in euros, and two are deliberate edge cases:
inv19 labels the total TOTAL instead of Total Due
inv20 starts with a chatty note before the invoice header
Edge cases are where prompt changes usually break, so give them their own tests rather than hoping the happy path covers them. As you find production failures, add a sanitized copy of each one to invoices/ and it becomes a permanent regression test.
A single "looks right" check isn't enough. The suite stacks four assertions on each test, from cheap and general to strict and specific.
1. Valid JSON matching a schema. Promptfoo's is-json assertion takes an optional JSON schema:
- type: is-json
value: file://schema.json
The schema requires every field, restricts currency to USD or EUR, requires a YYYY-MM-DD date, and needs at least one line item. That catches code fences, prose before the JSON, missing keys, and "$1,234.00" strings where numbers belong.
2. Exact field checks. For each invoice we know the right answer, so we assert it:
- type: javascript
value: |
const o = JSON.parse(output);
return o.invoice_number === 'INV-1007' && o.currency === 'EUR' && Math.abs(o.total - 1043.29) < 0.01;
This is the assertion that would have caught the euro bug in the opening story. Compare floats with a tolerance, never ===.
3. Internal consistency. A model can copy the total correctly and still invent a line item. asserts/totals-add-up.js recomputes the math:
module.exports = (output) => {
const o = JSON.parse(output);
const sum = o.line_items.reduce((s, l) => s + l.quantity * l.unit_price, 0);
const ok = Math.abs(sum - o.subtotal) < 0.01 && Math.abs(o.subtotal + o.tax - o.total) < 0.01;
return { pass: ok, score: ok ? 1 : 0,
reason: ok ? 'totals reconcile' : `sum ${sum.toFixed(2)} vs subtotal ${o.subtotal}` };
};
Returning {pass, score, reason} instead of a bare boolean means the Promptfoo viewer shows why a case failed, which saves a lot of time when you're debugging twenty red cells.
4. Rubric grading for fuzzy fields (optional). Some fields don't have one right string. "Acme Supplies" and "Acme Supplies Inc." are both acceptable vendor names. Promptfoo's llm-rubric assertion asks a grading model to judge:
- type: llm-rubric
value: The vendor field names the same company as the "From:" line, ignoring suffixes like Inc or LLC.
llm-rubric needs a real grading model and an API key, so the example repo leaves it off by default. Turn it on once you add a provider key (see Step 6).
Here's a practical problem: you want the pipeline in place on day one, but you may not want to put an API key in CI yet, or pay for evals on every documentation PR. Promptfoo supports custom providers, which are plain JavaScript classes with a callApi method, so the repo ships a deterministic stand-in:
class LocalExtractor {
id() { return 'local-extractor'; }
async callApi(prompt) {
const t = prompt.slice(prompt.indexOf('INVOICE'));
// ...regex extraction of number, vendor, date, line items, totals...
return { output: JSON.stringify(out) };
}
}
module.exports = LocalExtractor;
It's wired in with id: file://providers/local-extractor.js. To be clear, this doesn't measure model quality. It parses the invoice text out of the rendered prompt. What it does show is that your config, tests, schema, custom assertions and CI gate all work, for free, before any model is involved. It's also a useful baseline: if a real model scores below a regex parser on clean synthetic data, you'll want to know.
Running it locally:
npx promptfoo@0.125.0 eval -c promptfooconfig.yaml --no-cache
npx promptfoo@0.125.0 view
The output is a grid with one row per invoice and one column per prompt. With the local provider, all 40 cells (20 tests × 2 prompts) pass.
The workflow runs whenever a PR touches prompts, tests, invoices, assertions or providers:
on:
pull_request:
paths: ['prompts/**', 'promptfooconfig.yaml', 'tests.yaml', 'invoices/**', 'asserts/**', 'providers/**']
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22.22 }
- run: npx --yes promptfoo@0.125.0 eval -c promptfooconfig.yaml --no-cache -o results.json
env: { PROMPTFOO_DISABLE_TELEMETRY: 1 }
- name: Fail if pass rate < 95%
run: |
node -e "const r=require('./results.json').results.stats;
const rate=r.successes/(r.successes+r.failures);
console.log('pass rate',rate); if(rate<0.95) process.exit(1)"
- uses: actions/upload-artifact@v4
if: always()
with: { name: promptfoo-results, path: results.json }
A few design choices are worth calling out:
npx promptfoo will eventually change under you. Pin the model version too, once you add one.temperature: 0 for extraction tasks.results.json even on failure means reviewers can download the exact outputs that failed without re-running anything.--no-cache), so a passing cached result can't hide a regression after a provider-side model update.
When you're ready, uncomment a provider in promptfooconfig.yaml:
providers:
- id: file://providers/local-extractor.js
label: local-extractor
- id: openai:gpt-4.1-mini
config: { temperature: 0 }
Add the key as a repository secret (OPENAI_API_KEY) and pass it through the workflow's env. Keeping the local extractor alongside the real model gives you a stable baseline column in every run. Now llm-rubric can be turned on too.
On cost: 20 short invoices × 2 prompts × 1 model is 40 small requests per run, so with a mini-sized model a run costs well under a cent. If that adds up across many PRs, the paths: filter already skips runs for changes that can't affect prompt behavior.
promptfoo view locally, and filter to failures.reason string from the custom assertion. "sum 812.40 vs subtotal 912.40" points straight at a dropped line item.
Clone the repo, run npx promptfoo@0.125.0 eval, then try breaking prompts/v2.txt, for example by deleting the currency instruction, and watch the gate catch it.