Wire promptfoo into GitHub Actions so a bad prompt edit fails the pull request before it merges.
What you'll build #
A prompt regression suite that runs on every pull request: promptfoo scores your prompt against fixed test cases, and a GitHub Actions check fails the PR when an edit breaks tone, policy rules, or output shape. You'll know a prompt change is bad before it merges, not after a customer sees it.
Prerequisites #
Verified against promptfoo 0.123.0 (September 2026), promptfoo/promptfoo-action@v1, and Node.js 24 LTS.
- Node.js 22.22.0 or newer. promptfoo enforces this in its
enginesfield; 24 LTS is the safe choice. - A GitHub repository where you can add workflows and secrets.
- An Anthropic API key from the Anthropic Console . The example tests a Claude prompt, but promptfoo supports OpenAI, Google, Azure, and dozens of other providers; only the
providersline changes. - Commands are for macOS/Linux. On Windows, run them in WSL.
1. Scaffold the project #
mkdir prompt-ci && cd prompt-ci
git init
mkdir -p prompts .github/workflows
export ANTHROPIC_API_KEY=sk-ant-your-key
There's no install step. npx promptfoo@latest fetches the CLI on demand, which is also how the GitHub Action runs it.
2. Write the prompt and the test suite #
Create prompts/support-reply.txt. The rules in it are exactly what CI will defend:
You are a support agent for Acme Cloud. Write a reply to the customer ticket below.
Rules:
- Keep the reply under 120 words.
- Never promise a refund. Refund requests inside the 14-day window go to the billing team.
- Never mention internal tools, ticket queues, or escalation policies.
Ticket:
{{ticket}}
The {{ticket}} placeholder is a template variable that each test case fills in.
Now create promptfooconfig.yaml in the repo root:
description: Support reply prompt regression suite
prompts:
- file://prompts/support-reply.txt
providers:
- id: anthropic:messages:claude-opus-5
config:
max_tokens: 1024
defaultTest:
options:
provider: anthropic:messages:claude-opus-5
tests:
- description: angry refund request
vars:
ticket: "I bought the Pro plan three days ago and it broke my deploy pipeline. I want my money back NOW."
assert:
- type: icontains
value: billing
- type: not-icontains
value: I guarantee
- type: llm-rubric
value: Reply is professional, de-escalates, does not promise a refund, and directs the customer to the billing team.
- description: routine password reset
vars:
ticket: "How do I reset my password?"
assert:
- type: icontains
value: password
- type: javascript
value: output.split(/\s+/).length <= 160
Three things matter here. Deterministic assertions (icontains, not-icontains, javascript) are free and instant, so lead with them. llm-rubric sends the output to a grading model for the judgment calls a string match can't make, like "does this de-escalate". And defaultTest.options.provider pins that grader; without it, promptfoo picks a grading model from whatever credentials it finds in the environment, and you want CI runs judged by the same model every time.
3. Run the eval locally #
npx promptfoo@latest eval
Each test case runs against the prompt and every assertion is checked. Add npx promptfoo@latest view to browse results in a local web UI. The exit code is the CI contract: 0 when everything passes, 100 when at least one assertion fails, 1 for any other error.
4. Wire it into GitHub Actions #
Add the API key as a repository secret, with the GitHub CLI or under Settings > Secrets and variables > Actions:
gh secret set ANTHROPIC_API_KEY
Then create .github/workflows/prompt-eval.yml:
name: Prompt regression tests
on:
pull_request:
paths:
- 'prompts/**'
- 'promptfooconfig.yaml'
jobs:
evaluate:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: .promptfoo-cache
key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml') }}
- uses: promptfoo/promptfoo-action@v1
with:
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
github-token: ${{ secrets.GITHUB_TOKEN }}
config: promptfooconfig.yaml
prompts: 'prompts/**/*.txt'
cache-path: .promptfoo-cache
promptfoo-version: 0.123.0
fail-on-threshold: 100
The pieces: the paths filter skips the workflow when a PR doesn't touch prompts, so you're not paying for evals on unrelated changes. The cache step persists promptfoo's LLM response cache between runs; unchanged prompt-and-test pairs are replayed from disk instead of hitting the API. pull-requests: write lets the action post the results table as a PR comment. fail-on-threshold: 100 turns any assertion failure into a red check; lower it if you want a tolerance band instead of a hard gate. Pinning promptfoo-version keeps CI results reproducible.
By default the action evaluates the prompt files the PR actually changed, matched by the prompts glob. If you'd rather always run the full config as written, add use-config-prompts: true.
5. Break the prompt on purpose #
Prove the gate works:
git checkout -b loosen-refund-rule
sed -i '' '/Never promise a refund/d' prompts/support-reply.txt # GNU sed: drop the ''
git commit -am "Simplify support prompt"
git push -u origin loosen-refund-rule
gh pr create --fill
With the refund rule gone, the model tends to promise refunds to angry customers. The llm-rubric assertion catches it, the suite drops below 100%, and the check goes red with a comment showing which case failed and why.
Verify it works #
Locally, a passing run ends like this (table trimmed):
Running 2 test cases (up to 4 at a time)...
✓ Eval complete (ID: eval-xxx-2026-09-13T11:39:29)
» View results: promptfoo view
Results:
✓ 2 passed (100%)
0 failed (0%)
0 errors (0%)
Duration: 9s (concurrency: 4)
echo $? prints 0. In GitHub, a PR that touches prompts/ gets a "Prompt regression tests" check plus a bot comment with the pass/fail matrix, and the PR from step 5 shows the check failing.
Troubleshooting #
✗ Missing ANTHROPIC_API_KEY (anthropic:messages:claude-opus-5): promptfoo validates credentials before running anything. Locally, export ANTHROPIC_API_KEY=... in the same shell. In CI, the secret name in ${{ secrets.ANTHROPIC_API_KEY }} must match what you created, and the anthropic-api-key input must be present. Note that secrets aren't exposed to workflows triggered from forks.
npm warn EBADENGINE Unsupported engine followed by crashes: your Node is older than 22.22.0, which promptfoo 0.123.0 requires. Upgrade to 24 LTS.
Resource not accessible by integration in the action logs: the workflow can run the eval but can't post the PR comment. Add pull-requests: write under permissions: (a repo-level "Read repository contents" default token permission causes this too).
Local eval exits with code 100: that's by design, the documented exit code for "at least one test failed," not a crash. Read the failure rows in the output table, or set PROMPTFOO_FAILED_TEST_EXIT_CODE=0 if a script needs the run to be non-fatal.
Next steps #
Add cost and latency assertions so a prompt edit that doubles token spend or response time also fails CI. List a second entry under providers to score a model upgrade against your current one in the same table, which is the same regression mechanism pointed at model changes instead of prompt changes. Move test cases into a CSV once they outgrow the YAML. And when the suite is stable, look at promptfoo's red teaming to probe the same prompt for jailbreaks and data leaks on a schedule rather than per PR.
Sources & further reading #
-
[Getting started](https://www.promptfoo.dev/docs/getting-started/) — promptfoo.dev -
[Anthropic provider](https://www.promptfoo.dev/docs/providers/anthropic/) — promptfoo.dev -
[Command line reference](https://www.promptfoo.dev/docs/usage/command-line/) — promptfoo.dev -
[Deterministic assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/) — promptfoo.dev -
[Model-graded metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/) — promptfoo.dev -
[promptfoo GitHub Action README](https://github.com/promptfoo/promptfoo-action) — github.com
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 1 #
had to debug a production LLM call last month where someone tweaked the system prompt and suddenly it started returning json when it should've been plain text—would've caught that instantly with this. the nice part is promptfoo treats tests like actual assertions instead of vibes, which is exactly how i think about type checking in rust. definitely worth the 20 minutes to wire up.