# Regression-Test Your Prompts with promptfoo in CI > Source: > Published: 2026-09-07 11:41:58+00:00 # Regression-Test Your Prompts with promptfoo in CI Catch prompt and model regressions on every pull request with promptfoo evals in GitHub Actions. [Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein) ## What you'll build A regression-test suite for an LLM prompt using [promptfoo](https://www.promptfoo.dev), wired into GitHub Actions so every pull request that touches your prompts runs the evals, posts a pass/fail comment on the PR, and fails the check if a change breaks expected behavior. ## Prerequisites - Node.js >= 22.22.0. promptfoo enforces this hard, so use [Node.js](https://nodejs.org) 24 LTS. Verified with Node 24.1.0. - promptfoo 0.122.2 (the current release on npm; all commands below were run against it). No install needed if you use `npx` . - An Anthropic API key from the [Anthropic Console](https://console.anthropic.com) . Any provider promptfoo supports works; this tutorial uses Claude. - A GitHub repository with Actions enabled, and the [GitHub CLI](https://cli.github.com) (`gh` ) for adding the secret. - Commands assume a POSIX shell (macOS/Linux). Windows users: run them in Git Bash or WSL. ## 1. Set up the project From your repo root, create a directory for prompt files: ``` mkdir -p prompts export ANTHROPIC_API_KEY=sk-ant-... # your key, for local runs ``` You don't need to install anything. `npx promptfoo@latest` fetches the CLI on demand, and the GitHub Action installs its own copy in CI. ## 2. Write the prompt under test Create `prompts/triage.txt`. This is a support-ticket triage prompt with a strict output contract, which is exactly the kind of thing that silently breaks when someone "improves" the wording: ``` You are a support ticket triage assistant. Read the ticket below and respond with only a JSON object — no markdown, no code fences — with exactly these keys: - "category": one of "billing", "account", "bug", "other" - "urgency": one of "low", "medium", "high" - "summary": one neutral, professional sentence describing the issue Ticket: {{ticket}} ``` `{{ticket}}` is a Nunjucks variable. Each test case fills it in. ## 3. Configure the eval Create `promptfooconfig.yaml` in the repo root: ``` php # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json description: Support ticket triage regression tests prompts: - file://prompts/triage.txt providers: - id: anthropic:messages:claude-opus-5 config: max_tokens: 512 defaultTest: options: provider: anthropic:messages:claude-opus-5 tests: - description: duplicate charge routes to billing vars: ticket: I was charged twice for my subscription this month and need the duplicate refunded. assert: - type: is-json - type: javascript value: JSON.parse(output).category === 'billing' - description: password reset is low urgency vars: ticket: How do I reset my password? Not urgent, just locked out of the mobile app. assert: - type: is-json - type: javascript value: JSON.parse(output).urgency === 'low' - description: angry ticket still gets a neutral summary vars: ticket: This is the THIRD time your app deleted my data. Fix it or I'm cancelling and telling everyone. assert: - type: llm-rubric value: The summary is neutral and professional and does not adopt the customer's angry tone. ``` Three details matter here. The `javascript` assertions are inline expressions that get the raw model output as `output`, so you can parse and check exact fields deterministically and for free. The `llm-rubric` assertion uses a model as grader for the fuzzy tone requirement. And the `defaultTest.options.provider` block pins that grader to Claude, because promptfoo grades with OpenAI by default and you'd otherwise need a second API key in CI. The Anthropic provider runs at temperature 0 by default, which keeps reruns stable. ## 4. Run it locally ``` npx promptfoo@latest eval ``` `eval` auto-loads `promptfooconfig.yaml` from the current directory. Open the results table in a browser with: ``` npx promptfoo@latest view ``` The exit code is the CI contract: 0 when everything passes, 100 when at least one assertion fails, 1 on any other error. You'll rely on that in the next step. ## 5. Wire it into GitHub Actions Add your key as a repo secret: ``` gh secret set ANTHROPIC_API_KEY ``` Create `.github/workflows/prompt-eval.yml`: ``` name: Prompt regression tests on: pull_request: paths: - 'prompts/**' - 'promptfooconfig.yaml' jobs: evaluate: runs-on: ubuntu-latest permissions: contents: read pull-requests: write steps: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: node-version: 24 - name: Cache promptfoo uses: actions/cache@v6 with: path: ~/.cache/promptfoo key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml') }} restore-keys: | ${{ runner.os }}-promptfoo- - uses: promptfoo/promptfoo-action@v1 with: anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }} github-token: ${{ secrets.GITHUB_TOKEN }} config: promptfooconfig.yaml use-config-prompts: true cache-path: ~/.cache/promptfoo no-share: true ``` Why each piece: the `paths` filter skips the job on PRs that don't touch prompts or tests. `pull-requests: write` lets the action post its results comment. The cache stores LLM responses keyed on request content, so unchanged prompt/test pairs cost nothing on reruns. `use-config-prompts: true` makes the action evaluate the prompt list from your config instead of trying to infer it from changed files. `no-share: true` keeps eval results out of promptfoo's public sharing service. ## Verify it works Locally, a passing run ends like this (IDs and timing will differ): ``` Running 3 test cases (up to 4 at a time)... ✓ Eval complete (ID: eval-1Pr-2026-09-07T11:38:23) Results: ✓ 3 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 9s (concurrency: 4) ``` Confirm the exit code with `echo $?` (expect `0`). Then prove the CI gate actually gates. Open a branch, edit `prompts/triage.txt` to remove the line about responding "with only a JSON object", and open a PR. The workflow runs, the `is-json` assertions fail on fenced output, the action comments a pass/fail table on the PR, and the check goes red with exit code 100. Revert the edit and the check goes green. ## Troubleshooting `Required: >=22.22.0` followed by `Install a supported Node.js version and try again.` Your Node is too old for current promptfoo. Install Node 24 (`nvm install 24` or `brew install node`) and re-run. In CI this can't happen because the workflow pins `node-version: 24`. `✗ Missing ANTHROPIC_API_KEY (anthropic:messages:claude-opus-5)` means the provider can't find a key. Locally, `export ANTHROPIC_API_KEY=...` in the same shell. In CI, check that the secret exists (`gh secret list`) and that the workflow passes it via the `anthropic-api-key` input. Secret names are case-sensitive. The CI job fails with exit code 100 but no stack trace. That's not a crash. 100 is promptfoo's "at least one test failed" code. Read the results table in the job log or the PR comment to see which assertion failed. If you need a different code, set the `PROMPTFOO_FAILED_TEST_EXIT_CODE` environment variable. `Resource not accessible by integration` when posting the PR comment means the workflow token lacks write access. Make sure the job has `permissions: pull-requests: write`, as in the workflow above. PRs from forks get a read-only token by design; for public repos, expect the comment step to fail on fork PRs even though the eval itself runs. ## Next steps - Add a second entry under `providers:` to compare models side by side in one results table. This is how you test a model upgrade before committing to it. - Set `fail-on-threshold: 90` on the action to allow a pass rate above 90% instead of requiring perfection, useful once your suite grows past a handful of cases. - Emit machine-readable results with `-o results.junit.xml` (also`.json` ,`.html` ) and feed them to your CI's test reporting. - Read promptfoo's [CI/CD integration guide](https://www.promptfoo.dev/docs/integrations/ci-cd/) for GitLab/Jenkins equivalents, and the[assertions reference](https://www.promptfoo.dev/docs/configuration/expected-outputs/) for the full assertion catalog, including semantic similarity and JSON schema validation. ## Sources & further reading 1. [Getting started](https://www.promptfoo.dev/docs/getting-started/) — promptfoo.dev 2. [GitHub Actions integration](https://www.promptfoo.dev/docs/integrations/github-action/) — promptfoo.dev 3. [Anthropic provider](https://www.promptfoo.dev/docs/providers/anthropic/) — promptfoo.dev 4. [Command line reference](https://www.promptfoo.dev/docs/usage/command-line/) — promptfoo.dev 5. [Assertions and metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/) — promptfoo.dev 6. [promptfoo-action repository](https://github.com/promptfoo/promptfoo-action) — github.com [Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop. ## Discussion 0 No comments yet Be the first to weigh in.