{"slug": "catch-prompt-regressions-in-ci-with-promptfoo-and-github-actions", "title": "Catch Prompt Regressions in CI with promptfoo and GitHub Actions", "summary": "Promptfoo 0.123.0 can be wired into GitHub Actions via promptfoo/promptfoo-action@v1 to score prompts against fixed test cases on every pull request, failing the PR when an edit breaks tone, policy rules, or output shape. The setup requires Node.js 22.22.0 or newer (24 LTS recommended) and an Anthropic API key, with the eval returning exit code 0 on success, 100 when at least one assertion fails, and 1 for any other error. Deterministic assertions such as icontains, not-icontains, and javascript run free and instantly, while llm-rubric sends output to a pinned grading model for judgment calls.", "body_md": "# Catch Prompt Regressions in CI with promptfoo and GitHub Actions\n\nWire promptfoo into GitHub Actions so a bad prompt edit fails the pull request before it merges.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\n## What you'll build\n\nA prompt regression suite that runs on every pull request: [promptfoo](https://www.promptfoo.dev/) scores your prompt against fixed test cases, and a [GitHub Actions](https://docs.github.com/en/actions) check fails the PR when an edit breaks tone, policy rules, or output shape. You'll know a prompt change is bad before it merges, not after a customer sees it.\n\n## Prerequisites\n\nVerified against promptfoo 0.123.0 (September 2026), `promptfoo/promptfoo-action@v1`, and Node.js 24 LTS.\n\n- [Node.js](https://nodejs.org/) 22.22.0 or newer. promptfoo enforces this in its`engines` field; 24 LTS is the safe choice.\n- A GitHub repository where you can add workflows and secrets.\n- An Anthropic API key from the [Anthropic Console](https://console.anthropic.com/) . The example tests a Claude prompt, but promptfoo supports OpenAI, Google, Azure, and dozens of other providers; only the`providers` line changes.\n- Commands are for macOS/Linux. On Windows, run them in WSL.\n\n## 1. Scaffold the project\n\n```\nmkdir prompt-ci && cd prompt-ci\ngit init\nmkdir -p prompts .github/workflows\nexport ANTHROPIC_API_KEY=sk-ant-your-key\n```\n\nThere's no install step. `npx promptfoo@latest` fetches the CLI on demand, which is also how the GitHub Action runs it.\n\n## 2. Write the prompt and the test suite\n\nCreate `prompts/support-reply.txt`. The rules in it are exactly what CI will defend:\n\n```\nYou are a support agent for Acme Cloud. Write a reply to the customer ticket below.\n\nRules:\n- Keep the reply under 120 words.\n- Never promise a refund. Refund requests inside the 14-day window go to the billing team.\n- Never mention internal tools, ticket queues, or escalation policies.\n\nTicket:\n{{ticket}}\n```\n\nThe `{{ticket}}` placeholder is a template variable that each test case fills in.\n\nNow create `promptfooconfig.yaml` in the repo root:\n\n``` php\n# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json\ndescription: Support reply prompt regression suite\n\nprompts:\n  - file://prompts/support-reply.txt\n\nproviders:\n  - id: anthropic:messages:claude-opus-5\n    config:\n      max_tokens: 1024\n\ndefaultTest:\n  options:\n    provider: anthropic:messages:claude-opus-5\n\ntests:\n  - description: angry refund request\n    vars:\n      ticket: \"I bought the Pro plan three days ago and it broke my deploy pipeline. I want my money back NOW.\"\n    assert:\n      - type: icontains\n        value: billing\n      - type: not-icontains\n        value: I guarantee\n      - type: llm-rubric\n        value: Reply is professional, de-escalates, does not promise a refund, and directs the customer to the billing team.\n\n  - description: routine password reset\n    vars:\n      ticket: \"How do I reset my password?\"\n    assert:\n      - type: icontains\n        value: password\n      - type: javascript\n        value: output.split(/\\s+/).length <= 160\n```\n\nThree things matter here. Deterministic assertions (`icontains`, `not-icontains`, `javascript`) are free and instant, so lead with them. `llm-rubric` sends the output to a grading model for the judgment calls a string match can't make, like \"does this de-escalate\". And `defaultTest.options.provider` pins that grader; without it, promptfoo picks a grading model from whatever credentials it finds in the environment, and you want CI runs judged by the same model every time.\n\n## 3. Run the eval locally\n\n```\nnpx promptfoo@latest eval\n```\n\nEach test case runs against the prompt and every assertion is checked. Add `npx promptfoo@latest view` to browse results in a local web UI. The exit code is the CI contract: `0` when everything passes, `100` when at least one assertion fails, `1` for any other error.\n\n## 4. Wire it into GitHub Actions\n\nAdd the API key as a repository secret, with the [GitHub CLI](https://cli.github.com/) or under Settings > Secrets and variables > Actions:\n\n```\ngh secret set ANTHROPIC_API_KEY\n```\n\nThen create `.github/workflows/prompt-eval.yml`:\n\n```\nname: Prompt regression tests\non:\n  pull_request:\n    paths:\n      - 'prompts/**'\n      - 'promptfooconfig.yaml'\n\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    permissions:\n      contents: read\n      pull-requests: write\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/cache@v4\n        with:\n          path: .promptfoo-cache\n          key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml') }}\n      - uses: promptfoo/promptfoo-action@v1\n        with:\n          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}\n          github-token: ${{ secrets.GITHUB_TOKEN }}\n          config: promptfooconfig.yaml\n          prompts: 'prompts/**/*.txt'\n          cache-path: .promptfoo-cache\n          promptfoo-version: 0.123.0\n          fail-on-threshold: 100\n```\n\nThe pieces: the `paths` filter skips the workflow when a PR doesn't touch prompts, so you're not paying for evals on unrelated changes. The cache step persists promptfoo's LLM response cache between runs; unchanged prompt-and-test pairs are replayed from disk instead of hitting the API. `pull-requests: write` lets the action post the results table as a PR comment. `fail-on-threshold: 100` turns any assertion failure into a red check; lower it if you want a tolerance band instead of a hard gate. Pinning `promptfoo-version` keeps CI results reproducible.\n\nBy default the action evaluates the prompt files the PR actually changed, matched by the `prompts` glob. If you'd rather always run the full config as written, add `use-config-prompts: true`.\n\n## 5. Break the prompt on purpose\n\nProve the gate works:\n\n```\ngit checkout -b loosen-refund-rule\nsed -i '' '/Never promise a refund/d' prompts/support-reply.txt   # GNU sed: drop the ''\ngit commit -am \"Simplify support prompt\"\ngit push -u origin loosen-refund-rule\ngh pr create --fill\n```\n\nWith the refund rule gone, the model tends to promise refunds to angry customers. The `llm-rubric` assertion catches it, the suite drops below 100%, and the check goes red with a comment showing which case failed and why.\n\n## Verify it works\n\nLocally, a passing run ends like this (table trimmed):\n\n```\nRunning 2 test cases (up to 4 at a time)...\n✓ Eval complete (ID: eval-xxx-2026-09-13T11:39:29)\n\n» View results: promptfoo view\n\nResults:\n  ✓ 2 passed (100%)\n  0 failed (0%)\n  0 errors (0%)\nDuration: 9s (concurrency: 4)\n```\n\n`echo $?` prints `0`. In GitHub, a PR that touches `prompts/` gets a \"Prompt regression tests\" check plus a bot comment with the pass/fail matrix, and the PR from step 5 shows the check failing.\n\n## Troubleshooting\n\n**`✗ Missing ANTHROPIC_API_KEY (anthropic:messages:claude-opus-5)`**: promptfoo validates credentials before running anything. Locally, `export ANTHROPIC_API_KEY=...` in the same shell. In CI, the secret name in `${{ secrets.ANTHROPIC_API_KEY }}` must match what you created, and the `anthropic-api-key` input must be present. Note that secrets aren't exposed to workflows triggered from forks.\n\n**`npm warn EBADENGINE Unsupported engine`** followed by crashes: your Node is older than 22.22.0, which promptfoo 0.123.0 requires. Upgrade to 24 LTS.\n\n**`Resource not accessible by integration`** in the action logs: the workflow can run the eval but can't post the PR comment. Add `pull-requests: write` under `permissions:` (a repo-level \"Read repository contents\" default token permission causes this too).\n\n**Local `eval` exits with code 100**: that's by design, the documented exit code for \"at least one test failed,\" not a crash. Read the failure rows in the output table, or set `PROMPTFOO_FAILED_TEST_EXIT_CODE=0` if a script needs the run to be non-fatal.\n\n## Next steps\n\nAdd [`cost` and `latency` assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/) so a prompt edit that doubles token spend or response time also fails CI. List a second entry under `providers` to score a model upgrade against your current one in the same table, which is the same regression mechanism pointed at model changes instead of prompt changes. Move test cases into a CSV once they outgrow the YAML. And when the suite is stable, look at [promptfoo's red teaming](https://www.promptfoo.dev/docs/red-team/) to probe the same prompt for jailbreaks and data leaks on a schedule rather than per PR.\n\n## Sources & further reading\n\n1. \n                                    [Getting started](https://www.promptfoo.dev/docs/getting-started/)\n                                — promptfoo.dev\n2. \n                                    [Anthropic provider](https://www.promptfoo.dev/docs/providers/anthropic/)\n                                — promptfoo.dev\n3. \n                                    [Command line reference](https://www.promptfoo.dev/docs/usage/command-line/)\n                                — promptfoo.dev\n4. \n                                    [Deterministic assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/)\n                                — promptfoo.dev\n5. \n                                    [Model-graded metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/)\n                                — promptfoo.dev\n6. \n                                    [promptfoo GitHub Action README](https://github.com/promptfoo/promptfoo-action)\n                                — github.com\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 1\n\nhad to debug a production LLM call last month where someone tweaked the system prompt and suddenly it started returning json when it should've been plain text—would've caught that instantly with this. the nice part is promptfoo treats tests like actual assertions instead of vibes, which is exactly how i think about type checking in rust. definitely worth the 20 minutes to wire up.", "url": "https://wpnews.pro/news/catch-prompt-regressions-in-ci-with-promptfoo-and-github-actions", "canonical_source": "https://sourcefeed.dev/a/catch-prompt-regressions-in-ci-with-promptfoo-and-github-actions", "published_at": "2026-09-13 11:42:41+00:00", "updated_at": "2026-09-13 12:04:30.844334+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-products"], "entities": ["promptfoo", "GitHub Actions", "Node.js", "Anthropic", "Claude", "Mariana Souza", "Acme Cloud", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/catch-prompt-regressions-in-ci-with-promptfoo-and-github-actions", "markdown": "https://wpnews.pro/news/catch-prompt-regressions-in-ci-with-promptfoo-and-github-actions.md", "text": "https://wpnews.pro/news/catch-prompt-regressions-in-ci-with-promptfoo-and-github-actions.txt", "jsonld": "https://wpnews.pro/news/catch-prompt-regressions-in-ci-with-promptfoo-and-github-actions.jsonld"}}