# Catch Prompt Regressions in CI with promptfoo and GitHub Actions > Source: > Published: 2026-09-13 11:42:41+00:00 # Catch Prompt Regressions in CI with promptfoo and GitHub Actions Wire promptfoo into GitHub Actions so a bad prompt edit fails the pull request before it merges. [Mariana Souza](https://sourcefeed.dev/u/mariana_souza) ## What you'll build A prompt regression suite that runs on every pull request: [promptfoo](https://www.promptfoo.dev/) scores your prompt against fixed test cases, and a [GitHub Actions](https://docs.github.com/en/actions) check fails the PR when an edit breaks tone, policy rules, or output shape. You'll know a prompt change is bad before it merges, not after a customer sees it. ## Prerequisites Verified against promptfoo 0.123.0 (September 2026), `promptfoo/promptfoo-action@v1`, and Node.js 24 LTS. - [Node.js](https://nodejs.org/) 22.22.0 or newer. promptfoo enforces this in its`engines` field; 24 LTS is the safe choice. - A GitHub repository where you can add workflows and secrets. - An Anthropic API key from the [Anthropic Console](https://console.anthropic.com/) . The example tests a Claude prompt, but promptfoo supports OpenAI, Google, Azure, and dozens of other providers; only the`providers` line changes. - Commands are for macOS/Linux. On Windows, run them in WSL. ## 1. Scaffold the project ``` mkdir prompt-ci && cd prompt-ci git init mkdir -p prompts .github/workflows export ANTHROPIC_API_KEY=sk-ant-your-key ``` There's no install step. `npx promptfoo@latest` fetches the CLI on demand, which is also how the GitHub Action runs it. ## 2. Write the prompt and the test suite Create `prompts/support-reply.txt`. The rules in it are exactly what CI will defend: ``` You are a support agent for Acme Cloud. Write a reply to the customer ticket below. Rules: - Keep the reply under 120 words. - Never promise a refund. Refund requests inside the 14-day window go to the billing team. - Never mention internal tools, ticket queues, or escalation policies. Ticket: {{ticket}} ``` The `{{ticket}}` placeholder is a template variable that each test case fills in. Now create `promptfooconfig.yaml` in the repo root: ``` php # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json description: Support reply prompt regression suite prompts: - file://prompts/support-reply.txt providers: - id: anthropic:messages:claude-opus-5 config: max_tokens: 1024 defaultTest: options: provider: anthropic:messages:claude-opus-5 tests: - description: angry refund request vars: ticket: "I bought the Pro plan three days ago and it broke my deploy pipeline. I want my money back NOW." assert: - type: icontains value: billing - type: not-icontains value: I guarantee - type: llm-rubric value: Reply is professional, de-escalates, does not promise a refund, and directs the customer to the billing team. - description: routine password reset vars: ticket: "How do I reset my password?" assert: - type: icontains value: password - type: javascript value: output.split(/\s+/).length <= 160 ``` Three things matter here. Deterministic assertions (`icontains`, `not-icontains`, `javascript`) are free and instant, so lead with them. `llm-rubric` sends the output to a grading model for the judgment calls a string match can't make, like "does this de-escalate". And `defaultTest.options.provider` pins that grader; without it, promptfoo picks a grading model from whatever credentials it finds in the environment, and you want CI runs judged by the same model every time. ## 3. Run the eval locally ``` npx promptfoo@latest eval ``` Each test case runs against the prompt and every assertion is checked. Add `npx promptfoo@latest view` to browse results in a local web UI. The exit code is the CI contract: `0` when everything passes, `100` when at least one assertion fails, `1` for any other error. ## 4. Wire it into GitHub Actions Add the API key as a repository secret, with the [GitHub CLI](https://cli.github.com/) or under Settings > Secrets and variables > Actions: ``` gh secret set ANTHROPIC_API_KEY ``` Then create `.github/workflows/prompt-eval.yml`: ``` name: Prompt regression tests on: pull_request: paths: - 'prompts/**' - 'promptfooconfig.yaml' jobs: evaluate: runs-on: ubuntu-latest permissions: contents: read pull-requests: write steps: - uses: actions/checkout@v4 - uses: actions/cache@v4 with: path: .promptfoo-cache key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml') }} - uses: promptfoo/promptfoo-action@v1 with: anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }} github-token: ${{ secrets.GITHUB_TOKEN }} config: promptfooconfig.yaml prompts: 'prompts/**/*.txt' cache-path: .promptfoo-cache promptfoo-version: 0.123.0 fail-on-threshold: 100 ``` The pieces: the `paths` filter skips the workflow when a PR doesn't touch prompts, so you're not paying for evals on unrelated changes. The cache step persists promptfoo's LLM response cache between runs; unchanged prompt-and-test pairs are replayed from disk instead of hitting the API. `pull-requests: write` lets the action post the results table as a PR comment. `fail-on-threshold: 100` turns any assertion failure into a red check; lower it if you want a tolerance band instead of a hard gate. Pinning `promptfoo-version` keeps CI results reproducible. By default the action evaluates the prompt files the PR actually changed, matched by the `prompts` glob. If you'd rather always run the full config as written, add `use-config-prompts: true`. ## 5. Break the prompt on purpose Prove the gate works: ``` git checkout -b loosen-refund-rule sed -i '' '/Never promise a refund/d' prompts/support-reply.txt # GNU sed: drop the '' git commit -am "Simplify support prompt" git push -u origin loosen-refund-rule gh pr create --fill ``` With the refund rule gone, the model tends to promise refunds to angry customers. The `llm-rubric` assertion catches it, the suite drops below 100%, and the check goes red with a comment showing which case failed and why. ## Verify it works Locally, a passing run ends like this (table trimmed): ``` Running 2 test cases (up to 4 at a time)... ✓ Eval complete (ID: eval-xxx-2026-09-13T11:39:29) » View results: promptfoo view Results: ✓ 2 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 9s (concurrency: 4) ``` `echo $?` prints `0`. In GitHub, a PR that touches `prompts/` gets a "Prompt regression tests" check plus a bot comment with the pass/fail matrix, and the PR from step 5 shows the check failing. ## Troubleshooting **`✗ Missing ANTHROPIC_API_KEY (anthropic:messages:claude-opus-5)`**: promptfoo validates credentials before running anything. Locally, `export ANTHROPIC_API_KEY=...` in the same shell. In CI, the secret name in `${{ secrets.ANTHROPIC_API_KEY }}` must match what you created, and the `anthropic-api-key` input must be present. Note that secrets aren't exposed to workflows triggered from forks. **`npm warn EBADENGINE Unsupported engine`** followed by crashes: your Node is older than 22.22.0, which promptfoo 0.123.0 requires. Upgrade to 24 LTS. **`Resource not accessible by integration`** in the action logs: the workflow can run the eval but can't post the PR comment. Add `pull-requests: write` under `permissions:` (a repo-level "Read repository contents" default token permission causes this too). **Local `eval` exits with code 100**: that's by design, the documented exit code for "at least one test failed," not a crash. Read the failure rows in the output table, or set `PROMPTFOO_FAILED_TEST_EXIT_CODE=0` if a script needs the run to be non-fatal. ## Next steps Add [`cost` and `latency` assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/) so a prompt edit that doubles token spend or response time also fails CI. List a second entry under `providers` to score a model upgrade against your current one in the same table, which is the same regression mechanism pointed at model changes instead of prompt changes. Move test cases into a CSV once they outgrow the YAML. And when the suite is stable, look at [promptfoo's red teaming](https://www.promptfoo.dev/docs/red-team/) to probe the same prompt for jailbreaks and data leaks on a schedule rather than per PR. ## Sources & further reading 1. [Getting started](https://www.promptfoo.dev/docs/getting-started/) — promptfoo.dev 2. [Anthropic provider](https://www.promptfoo.dev/docs/providers/anthropic/) — promptfoo.dev 3. [Command line reference](https://www.promptfoo.dev/docs/usage/command-line/) — promptfoo.dev 4. [Deterministic assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/) — promptfoo.dev 5. [Model-graded metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/) — promptfoo.dev 6. [promptfoo GitHub Action README](https://github.com/promptfoo/promptfoo-action) — github.com [Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. ## Discussion 1 had to debug a production LLM call last month where someone tweaked the system prompt and suddenly it started returning json when it should've been plain text—would've caught that instantly with this. the nice part is promptfoo treats tests like actual assertions instead of vibes, which is exactly how i think about type checking in rust. definitely worth the 20 minutes to wire up.