Catch LLM Regressions in CI with promptfoo

Promptfoo 0.122.2 can be wired into GitHub Actions to run an eval suite against a golden dataset on every pull request that touches prompts, failing the CI check when outputs regress and posting a before/after comparison as a PR comment. The tutorial requires Node.js 22.22.0 or newer (Node 24 LTS recommended), an OpenAI API key, and supports 60+ providers including Anthropic, Azure, and local models; `promptfoo eval` exits with code 100 when any test fails, which fails the CI job. The setup uses a support-ticket triage prompt returning strict JSON with a `defaultTest` is-json assertion plus per-case javascript assertions for category and urgency.

Catch LLM Regressions in CI with promptfoo Wire promptfoo into GitHub Actions so prompt changes get tested against a golden dataset before merge. Priya Nair https://sourcefeed.dev/u/priya nair What you'll build A GitHub Actions check that runs a promptfoo https://www.promptfoo.dev eval suite against a golden dataset on every pull request that touches your prompts, fails the check when outputs regress, and posts a before/after comparison as a PR comment. Prerequisites - Node.js 22.22.0 or newer. promptfoo enforces this; the project recommends Node 24 LTS. Check with node --version . - promptfoo 0.122.2 the version this tutorial was verified against . Anything in the 0.122.x line works the same way. - A GitHub repository you can add secrets and workflows to. - An OpenAI API key. promptfoo supports 60+ providers, so swap in Anthropic, Azure, or a local model if you prefer; the action has a matching -api-key input for each. - Commands are written for macOS/Linux. On Windows, use WSL. 1. Install promptfoo and scaffold the project npm install -g promptfoo promptfoo --version 0.122.2 at time of writing mkdir prompt-ci && cd prompt-ci git init mkdir prompts You can also run everything through npx promptfoo@latest without installing, which is exactly what the GitHub Action does on the runner. 2. Write the prompt under test The example is a support-ticket triage prompt that must return strict JSON. Structured output is where silent regressions hurt most: a "small wording tweak" that makes the model chatty breaks every downstream parser. Create prompts/triage.txt : You are a support ticket triage bot for a SaaS product. Classify the ticket into exactly one category: billing, bug, account, or feature request. Rate urgency as low, medium, or high. Respond with JSON only, no prose: {"category": "...", "urgency": "..."} Ticket: {{ticket}} {{ticket}} is a Nunjucks https://mozilla.github.io/nunjucks/ variable that promptfoo fills from each test case. 3. Define the golden dataset Create promptfooconfig.yaml in the repo root. The tests block is your golden dataset: real inputs with known-correct expectations. Start with cases that have burned you before. php yaml-language-server: $schema=https://promptfoo.dev/config-schema.json description: Ticket triage regression suite prompts: - file://prompts/triage.txt providers: - id: openai:chat:gpt-5.6-luna config: reasoning effort: low max completion tokens: 200 defaultTest: assert: - type: is-json tests: - vars: ticket: I was charged twice for my subscription this month. assert: - type: javascript value: JSON.parse output .category === 'billing' - vars: ticket: The export button crashes the app and I have a demo in an hour assert: - type: javascript value: | const o = JSON.parse output ; return o.category === 'bug' && o.urgency === 'high'; - vars: ticket: Would love a dark mode someday, no rush. assert: - type: javascript value: | const o = JSON.parse output ; return o.category === 'feature request' && o.urgency === 'low'; defaultTest applies the is-json assertion to every case, so each test checks parseability plus its own field-level expectations. Deterministic assertions is-json , contains , regex , javascript are free and fast; save model-graded ones like llm-rubric for later. 4. Run the suite locally export OPENAI API KEY=sk-your-key promptfoo eval eval runs every prompt × provider × test combination and exits with code 100 if any test fails code 1 is reserved for operational errors like bad config . That exit code is the whole CI trick: a failing eval fails the job, same as a broken unit test. Run promptfoo view to inspect results in a local web UI. 5. Add the GitHub Actions workflow Store the API key as a repo secret first: gh secret set OPENAI API KEY Then create .github/workflows/promptfoo.yml using the official promptfoo-action https://github.com/promptfoo/promptfoo-action : name: Prompt evaluation on: pull request: paths: - 'prompts/ ' - 'promptfooconfig.yaml' jobs: evaluate: runs-on: ubuntu-latest permissions: contents: read pull-requests: write steps: - uses: actions/checkout@v4 - name: Cache promptfoo results uses: actions/cache@v4 with: path: .promptfoo-cache key: ${{ runner.os }}-promptfoo-${{ hashFiles 'promptfooconfig.yaml', 'prompts/ ' }} restore-keys: | ${{ runner.os }}-promptfoo- - name: Run promptfoo eval uses: promptfoo/promptfoo-action@v1 with: github-token: ${{ secrets.GITHUB TOKEN }} openai-api-key: ${{ secrets.OPENAI API KEY }} config: promptfooconfig.yaml prompts: 'prompts/ ' cache-path: .promptfoo-cache fail-on-threshold: 100 Three details matter here. The paths filter means the job only spends API tokens when prompt files or the config actually change. pull-requests: write lets the action post its results comment; without it the eval runs but the comment silently fails. fail-on-threshold: 100 fails the check unless every test passes; lower it say 90 once your suite is big enough that one flaky case shouldn't block a merge. The cache step keeps identical prompt+input pairs from being re-billed across runs. Commit everything and push: git add . git commit -m "Add promptfoo regression suite" git push origin main Verify it works Locally, a passing run ends like this table trimmed : ┌──────────────────────────┬──────────────────────────────┐ │ ticket │ openai:chat:gpt-5.6-luna │ ├──────────────────────────┼──────────────────────────────┤ │ I was charged twice for… │ PASS {"category":"billing"…│ └──────────────────────────┴──────────────────────────────┘ Successes: 3 Failures: 0 Errors: 0 Pass Rate: 100.00% echo $? prints 0 . Now prove the gate catches a regression. Branch off, edit prompts/triage.txt to end with Explain your reasoning, then give the JSON. a realistic "improvement" that breaks JSON-only output , and open a PR. Within a few minutes you should see the Prompt evaluation check go red, and a bot comment on the PR summarizing pass/fail counts with a link to the before/after comparison in the web viewer. Revert the line and the check goes green. Troubleshooting OpenAI API key is not set Set the OPENAI API KEY environment variable or add apiKey to the provider config. Locally, export it in the same shell running eval . In CI, the secret name in secrets.OPENAI API KEY must match what you created with gh secret set , and it has to be passed through the openai-api-key action input. Note that model-graded assertions like similar and llm-rubric call OpenAI as the default grader, so this error can appear even when your tested provider is a different vendor. Resource not accessible by integration in the action logs. The workflow token can't post the PR comment. Add the permissions block from step 5. On PRs from forks, GITHUB TOKEN is read-only by design; set disable-comment: true for fork PRs and rely on the check status alone. The check passes instantly without calling the model. The action evaluates prompt files changed in the PR. If your edit only touched promptfooconfig.yaml , or your prompts glob doesn't match the changed paths, nothing gets evaluated. Fix the glob, or set use-config-prompts: true to always use the prompts listed in the config, or force-run: true to evaluate regardless of the diff. npm warn EBADENGINE Unsupported engine followed by a crash. Your Node is older than 22.22.0. Upgrade to Node 24 LTS, or pin the runner with actions/setup-node before the promptfoo step. Next steps Move the golden dataset out of YAML once it grows: tests: file://tests.csv loads cases from a spreadsheet your support team can edit. Add model-graded checks llm-rubric , similar , factuality for prompts without a single right answer, and latency and cost assertions to catch performance regressions, not just quality ones. Since providers live in config, the same suite also gates model upgrades: add openai:chat:gpt-5.6-terra as a second provider and the PR comment becomes a side-by-side model comparison. The CI/CD integration docs https://www.promptfoo.dev/docs/integrations/ci-cd/ cover GitLab, Jenkins, and bare-CLI setups if you're not on GitHub Actions. Sources & further reading 1. Getting started https://www.promptfoo.dev/docs/getting-started/ — promptfoo.dev 2. GitHub Actions integration https://www.promptfoo.dev/docs/integrations/github-action/ — promptfoo.dev 3. promptfoo-action README https://github.com/promptfoo/promptfoo-action — github.com 4. Assertions and metrics reference https://www.promptfoo.dev/docs/configuration/expected-outputs/ — promptfoo.dev 5. Command line reference https://www.promptfoo.dev/docs/usage/command-line/ — promptfoo.dev 6. Troubleshooting https://www.promptfoo.dev/docs/usage/troubleshooting/ — promptfoo.dev Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 1 Does promptfoo handle cost tracking across runs, or do you end up just eating the bill every time someone tweaks a prompt in a PR? Running evals on every commit sounds amazing until you're suddenly explaining the AWS bill to finance.