{"slug": "catch-llm-regressions-in-ci-with-promptfoo", "title": "Catch LLM Regressions in CI with promptfoo", "summary": "Promptfoo 0.122.2 can be wired into GitHub Actions to run an eval suite against a golden dataset on every pull request that touches prompts, failing the CI check when outputs regress and posting a before/after comparison as a PR comment. The tutorial requires Node.js 22.22.0 or newer (Node 24 LTS recommended), an OpenAI API key, and supports 60+ providers including Anthropic, Azure, and local models; `promptfoo eval` exits with code 100 when any test fails, which fails the CI job. The setup uses a support-ticket triage prompt returning strict JSON with a `defaultTest` is-json assertion plus per-case javascript assertions for category and urgency.", "body_md": "# Catch LLM Regressions in CI with promptfoo\n\nWire promptfoo into GitHub Actions so prompt changes get tested against a golden dataset before merge.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\n## What you'll build\n\nA GitHub Actions check that runs a [promptfoo](https://www.promptfoo.dev) eval suite against a golden dataset on every pull request that touches your prompts, fails the check when outputs regress, and posts a before/after comparison as a PR comment.\n\n## Prerequisites\n\n- Node.js 22.22.0 or newer. promptfoo enforces this; the project recommends Node 24 LTS. Check with `node --version` .\n- promptfoo 0.122.2 (the version this tutorial was verified against). Anything in the 0.122.x line works the same way.\n- A GitHub repository you can add secrets and workflows to.\n- An OpenAI API key. promptfoo supports 60+ providers, so swap in Anthropic, Azure, or a local model if you prefer; the action has a matching `*-api-key` input for each.\n- Commands are written for macOS/Linux. On Windows, use WSL.\n\n## 1. Install promptfoo and scaffold the project\n\n```\nnpm install -g promptfoo\npromptfoo --version   # 0.122.2 at time of writing\nmkdir prompt-ci && cd prompt-ci\ngit init\nmkdir prompts\n```\n\nYou can also run everything through `npx promptfoo@latest` without installing, which is exactly what the GitHub Action does on the runner.\n\n## 2. Write the prompt under test\n\nThe example is a support-ticket triage prompt that must return strict JSON. Structured output is where silent regressions hurt most: a \"small wording tweak\" that makes the model chatty breaks every downstream parser.\n\nCreate `prompts/triage.txt`:\n\n```\nYou are a support ticket triage bot for a SaaS product.\nClassify the ticket into exactly one category: billing, bug, account, or feature_request.\nRate urgency as low, medium, or high.\nRespond with JSON only, no prose: {\"category\": \"...\", \"urgency\": \"...\"}\n\nTicket: {{ticket}}\n```\n\n`{{ticket}}` is a [Nunjucks](https://mozilla.github.io/nunjucks/) variable that promptfoo fills from each test case.\n\n## 3. Define the golden dataset\n\nCreate `promptfooconfig.yaml` in the repo root. The `tests` block is your golden dataset: real inputs with known-correct expectations. Start with cases that have burned you before.\n\n``` php\n# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json\ndescription: Ticket triage regression suite\n\nprompts:\n  - file://prompts/triage.txt\n\nproviders:\n  - id: openai:chat:gpt-5.6-luna\n    config:\n      reasoning_effort: low\n      max_completion_tokens: 200\n\ndefaultTest:\n  assert:\n    - type: is-json\n\ntests:\n  - vars:\n      ticket: I was charged twice for my subscription this month.\n    assert:\n      - type: javascript\n        value: JSON.parse(output).category === 'billing'\n  - vars:\n      ticket: The export button crashes the app and I have a demo in an hour!\n    assert:\n      - type: javascript\n        value: |\n          const o = JSON.parse(output);\n          return o.category === 'bug' && o.urgency === 'high';\n  - vars:\n      ticket: Would love a dark mode someday, no rush.\n    assert:\n      - type: javascript\n        value: |\n          const o = JSON.parse(output);\n          return o.category === 'feature_request' && o.urgency === 'low';\n```\n\n`defaultTest` applies the `is-json` assertion to every case, so each test checks parseability plus its own field-level expectations. Deterministic assertions (`is-json`, `contains`, `regex`, `javascript`) are free and fast; save model-graded ones like `llm-rubric` for later.\n\n## 4. Run the suite locally\n\n```\nexport OPENAI_API_KEY=sk-your-key\npromptfoo eval\n```\n\n`eval` runs every prompt × provider × test combination and exits with code `100` if any test fails (code `1` is reserved for operational errors like bad config). That exit code is the whole CI trick: a failing eval fails the job, same as a broken unit test. Run `promptfoo view` to inspect results in a local web UI.\n\n## 5. Add the GitHub Actions workflow\n\nStore the API key as a repo secret first:\n\n```\ngh secret set OPENAI_API_KEY\n```\n\nThen create `.github/workflows/promptfoo.yml` using the official [promptfoo-action](https://github.com/promptfoo/promptfoo-action):\n\n```\nname: Prompt evaluation\non:\n  pull_request:\n    paths:\n      - 'prompts/**'\n      - 'promptfooconfig.yaml'\n\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    permissions:\n      contents: read\n      pull-requests: write\n    steps:\n      - uses: actions/checkout@v4\n\n      - name: Cache promptfoo results\n        uses: actions/cache@v4\n        with:\n          path: .promptfoo-cache\n          key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml', 'prompts/**') }}\n          restore-keys: |\n            ${{ runner.os }}-promptfoo-\n\n      - name: Run promptfoo eval\n        uses: promptfoo/promptfoo-action@v1\n        with:\n          github-token: ${{ secrets.GITHUB_TOKEN }}\n          openai-api-key: ${{ secrets.OPENAI_API_KEY }}\n          config: promptfooconfig.yaml\n          prompts: 'prompts/**'\n          cache-path: .promptfoo-cache\n          fail-on-threshold: 100\n```\n\nThree details matter here. The `paths` filter means the job only spends API tokens when prompt files or the config actually change. `pull-requests: write` lets the action post its results comment; without it the eval runs but the comment silently fails. `fail-on-threshold: 100` fails the check unless every test passes; lower it (say `90`) once your suite is big enough that one flaky case shouldn't block a merge. The cache step keeps identical prompt+input pairs from being re-billed across runs.\n\nCommit everything and push:\n\n```\ngit add .\ngit commit -m \"Add promptfoo regression suite\"\ngit push origin main\n```\n\n## Verify it works\n\nLocally, a passing run ends like this (table trimmed):\n\n```\n┌──────────────────────────┬──────────────────────────────┐\n│ ticket                   │ [openai:chat:gpt-5.6-luna]   │\n├──────────────────────────┼──────────────────────────────┤\n│ I was charged twice for… │ [PASS] {\"category\":\"billing\"…│\n└──────────────────────────┴──────────────────────────────┘\nSuccesses: 3\nFailures: 0\nErrors: 0\nPass Rate: 100.00%\n```\n\n`echo $?` prints `0`. Now prove the gate catches a regression. Branch off, edit `prompts/triage.txt` to end with `Explain your reasoning, then give the JSON.` (a realistic \"improvement\" that breaks JSON-only output), and open a PR. Within a few minutes you should see the **Prompt evaluation** check go red, and a bot comment on the PR summarizing pass/fail counts with a link to the before/after comparison in the web viewer. Revert the line and the check goes green.\n\n## Troubleshooting\n\n**`OpenAI API key is not set`** Set the `OPENAI_API_KEY` environment variable or add `apiKey` to the provider config. Locally, export it in the same shell running `eval`. In CI, the secret name in `secrets.OPENAI_API_KEY` must match what you created with `gh secret set`, and it has to be passed through the `openai-api-key` action input. Note that model-graded assertions like `similar` and `llm-rubric` call OpenAI as the default grader, so this error can appear even when your tested provider is a different vendor.\n\n**`Resource not accessible by integration`** in the action logs. The workflow token can't post the PR comment. Add the `permissions` block from step 5. On PRs from forks, `GITHUB_TOKEN` is read-only by design; set `disable-comment: true` for fork PRs and rely on the check status alone.\n\n**The check passes instantly without calling the model.** The action evaluates prompt files changed in the PR. If your edit only touched `promptfooconfig.yaml`, or your `prompts` glob doesn't match the changed paths, nothing gets evaluated. Fix the glob, or set `use-config-prompts: true` to always use the prompts listed in the config, or `force-run: true` to evaluate regardless of the diff.\n\n**`npm warn EBADENGINE Unsupported engine`** followed by a crash. Your Node is older than 22.22.0. Upgrade to Node 24 LTS, or pin the runner with `actions/setup-node` before the promptfoo step.\n\n## Next steps\n\nMove the golden dataset out of YAML once it grows: `tests: file://tests.csv` loads cases from a spreadsheet your support team can edit. Add model-graded checks (`llm-rubric`, `similar`, `factuality`) for prompts without a single right answer, and `latency` and `cost` assertions to catch performance regressions, not just quality ones. Since providers live in config, the same suite also gates model upgrades: add `openai:chat:gpt-5.6-terra` as a second provider and the PR comment becomes a side-by-side model comparison. The [CI/CD integration docs](https://www.promptfoo.dev/docs/integrations/ci-cd/) cover GitLab, Jenkins, and bare-CLI setups if you're not on GitHub Actions.\n\n## Sources & further reading\n\n1. \n                                    [Getting started](https://www.promptfoo.dev/docs/getting-started/)\n                                — promptfoo.dev\n2. \n                                    [GitHub Actions integration](https://www.promptfoo.dev/docs/integrations/github-action/)\n                                — promptfoo.dev\n3. \n                                    [promptfoo-action README](https://github.com/promptfoo/promptfoo-action)\n                                — github.com\n4. \n                                    [Assertions and metrics reference](https://www.promptfoo.dev/docs/configuration/expected-outputs/)\n                                — promptfoo.dev\n5. \n                                    [Command line reference](https://www.promptfoo.dev/docs/usage/command-line/)\n                                — promptfoo.dev\n6. \n                                    [Troubleshooting](https://www.promptfoo.dev/docs/usage/troubleshooting/)\n                                — promptfoo.dev\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 1\n\nDoes promptfoo handle cost tracking across runs, or do you end up just eating the bill every time someone tweaks a prompt in a PR? Running evals on every commit sounds amazing until you're suddenly explaining the AWS bill to finance.", "url": "https://wpnews.pro/news/catch-llm-regressions-in-ci-with-promptfoo", "canonical_source": "https://sourcefeed.dev/a/catch-llm-regressions-in-ci-with-promptfoo", "published_at": "2026-09-12 11:39:25+00:00", "updated_at": "2026-09-12 12:09:34.842701+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "mlops", "ai-products"], "entities": ["promptfoo", "GitHub Actions", "Node.js", "OpenAI", "Anthropic", "Azure", "Priya Nair", "Nunjucks"], "alternates": {"html": "https://wpnews.pro/news/catch-llm-regressions-in-ci-with-promptfoo", "markdown": "https://wpnews.pro/news/catch-llm-regressions-in-ci-with-promptfoo.md", "text": "https://wpnews.pro/news/catch-llm-regressions-in-ci-with-promptfoo.txt", "jsonld": "https://wpnews.pro/news/catch-llm-regressions-in-ci-with-promptfoo.jsonld"}}