{"slug": "regression-test-your-prompts-with-promptfoo-in-ci", "title": "Regression-Test Your Prompts with promptfoo in CI", "summary": "A tutorial by Rachel Goldstein demonstrates how to regression-test LLM prompts using promptfoo 0.122.2 in GitHub Actions, catching prompt and model regressions on every pull request. The setup includes a support-ticket triage prompt with strict output contracts, evaluated via deterministic JavaScript assertions and an llm-rubric grader, and requires Node.js >= 22.22.0 and an Anthropic API key.", "body_md": "# Regression-Test Your Prompts with promptfoo in CI\n\nCatch prompt and model regressions on every pull request with promptfoo evals in GitHub Actions.\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)\n\n## What you'll build\n\nA regression-test suite for an LLM prompt using [promptfoo](https://www.promptfoo.dev), wired into GitHub Actions so every pull request that touches your prompts runs the evals, posts a pass/fail comment on the PR, and fails the check if a change breaks expected behavior.\n\n## Prerequisites\n\n- Node.js >= 22.22.0. promptfoo enforces this hard, so use [Node.js](https://nodejs.org) 24 LTS. Verified with Node 24.1.0.\n- promptfoo 0.122.2 (the current release on npm; all commands below were run against it). No install needed if you use `npx` .\n- An Anthropic API key from the [Anthropic Console](https://console.anthropic.com) . Any provider promptfoo supports works; this tutorial uses Claude.\n- A GitHub repository with Actions enabled, and the [GitHub CLI](https://cli.github.com) (`gh` ) for adding the secret.\n- Commands assume a POSIX shell (macOS/Linux). Windows users: run them in Git Bash or WSL.\n\n## 1. Set up the project\n\nFrom your repo root, create a directory for prompt files:\n\n```\nmkdir -p prompts\nexport ANTHROPIC_API_KEY=sk-ant-...   # your key, for local runs\n```\n\nYou don't need to install anything. `npx promptfoo@latest` fetches the CLI on demand, and the GitHub Action installs its own copy in CI.\n\n## 2. Write the prompt under test\n\nCreate `prompts/triage.txt`. This is a support-ticket triage prompt with a strict output contract, which is exactly the kind of thing that silently breaks when someone \"improves\" the wording:\n\n```\nYou are a support ticket triage assistant. Read the ticket below and respond with only a JSON object — no markdown, no code fences — with exactly these keys:\n- \"category\": one of \"billing\", \"account\", \"bug\", \"other\"\n- \"urgency\": one of \"low\", \"medium\", \"high\"\n- \"summary\": one neutral, professional sentence describing the issue\n\nTicket:\n{{ticket}}\n```\n\n`{{ticket}}` is a Nunjucks variable. Each test case fills it in.\n\n## 3. Configure the eval\n\nCreate `promptfooconfig.yaml` in the repo root:\n\n``` php\n# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json\ndescription: Support ticket triage regression tests\n\nprompts:\n  - file://prompts/triage.txt\n\nproviders:\n  - id: anthropic:messages:claude-opus-5\n    config:\n      max_tokens: 512\n\ndefaultTest:\n  options:\n    provider: anthropic:messages:claude-opus-5\n\ntests:\n  - description: duplicate charge routes to billing\n    vars:\n      ticket: I was charged twice for my subscription this month and need the duplicate refunded.\n    assert:\n      - type: is-json\n      - type: javascript\n        value: JSON.parse(output).category === 'billing'\n\n  - description: password reset is low urgency\n    vars:\n      ticket: How do I reset my password? Not urgent, just locked out of the mobile app.\n    assert:\n      - type: is-json\n      - type: javascript\n        value: JSON.parse(output).urgency === 'low'\n\n  - description: angry ticket still gets a neutral summary\n    vars:\n      ticket: This is the THIRD time your app deleted my data. Fix it or I'm cancelling and telling everyone.\n    assert:\n      - type: llm-rubric\n        value: The summary is neutral and professional and does not adopt the customer's angry tone.\n```\n\nThree details matter here. The `javascript` assertions are inline expressions that get the raw model output as `output`, so you can parse and check exact fields deterministically and for free. The `llm-rubric` assertion uses a model as grader for the fuzzy tone requirement. And the `defaultTest.options.provider` block pins that grader to Claude, because promptfoo grades with OpenAI by default and you'd otherwise need a second API key in CI. The Anthropic provider runs at temperature 0 by default, which keeps reruns stable.\n\n## 4. Run it locally\n\n```\nnpx promptfoo@latest eval\n```\n\n`eval` auto-loads `promptfooconfig.yaml` from the current directory. Open the results table in a browser with:\n\n```\nnpx promptfoo@latest view\n```\n\nThe exit code is the CI contract: 0 when everything passes, 100 when at least one assertion fails, 1 on any other error. You'll rely on that in the next step.\n\n## 5. Wire it into GitHub Actions\n\nAdd your key as a repo secret:\n\n```\ngh secret set ANTHROPIC_API_KEY\n```\n\nCreate `.github/workflows/prompt-eval.yml`:\n\n```\nname: Prompt regression tests\n\non:\n  pull_request:\n    paths:\n      - 'prompts/**'\n      - 'promptfooconfig.yaml'\n\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    permissions:\n      contents: read\n      pull-requests: write\n    steps:\n      - uses: actions/checkout@v7\n\n      - uses: actions/setup-node@v7\n        with:\n          node-version: 24\n\n      - name: Cache promptfoo\n        uses: actions/cache@v6\n        with:\n          path: ~/.cache/promptfoo\n          key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml') }}\n          restore-keys: |\n            ${{ runner.os }}-promptfoo-\n\n      - uses: promptfoo/promptfoo-action@v1\n        with:\n          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}\n          github-token: ${{ secrets.GITHUB_TOKEN }}\n          config: promptfooconfig.yaml\n          use-config-prompts: true\n          cache-path: ~/.cache/promptfoo\n          no-share: true\n```\n\nWhy each piece: the `paths` filter skips the job on PRs that don't touch prompts or tests. `pull-requests: write` lets the action post its results comment. The cache stores LLM responses keyed on request content, so unchanged prompt/test pairs cost nothing on reruns. `use-config-prompts: true` makes the action evaluate the prompt list from your config instead of trying to infer it from changed files. `no-share: true` keeps eval results out of promptfoo's public sharing service.\n\n## Verify it works\n\nLocally, a passing run ends like this (IDs and timing will differ):\n\n```\nRunning 3 test cases (up to 4 at a time)...\n✓ Eval complete (ID: eval-1Pr-2026-09-07T11:38:23)\n\nResults:\n  ✓ 3 passed (100%)\n  0 failed (0%)\n  0 errors (0%)\nDuration: 9s (concurrency: 4)\n```\n\nConfirm the exit code with `echo $?` (expect `0`).\n\nThen prove the CI gate actually gates. Open a branch, edit `prompts/triage.txt` to remove the line about responding \"with only a JSON object\", and open a PR. The workflow runs, the `is-json` assertions fail on fenced output, the action comments a pass/fail table on the PR, and the check goes red with exit code 100. Revert the edit and the check goes green.\n\n## Troubleshooting\n\n`Required: >=22.22.0` followed by `Install a supported Node.js version and try again.` Your Node is too old for current promptfoo. Install Node 24 (`nvm install 24` or `brew install node`) and re-run. In CI this can't happen because the workflow pins `node-version: 24`.\n\n`✗ Missing ANTHROPIC_API_KEY (anthropic:messages:claude-opus-5)` means the provider can't find a key. Locally, `export ANTHROPIC_API_KEY=...` in the same shell. In CI, check that the secret exists (`gh secret list`) and that the workflow passes it via the `anthropic-api-key` input. Secret names are case-sensitive.\n\nThe CI job fails with exit code 100 but no stack trace. That's not a crash. 100 is promptfoo's \"at least one test failed\" code. Read the results table in the job log or the PR comment to see which assertion failed. If you need a different code, set the `PROMPTFOO_FAILED_TEST_EXIT_CODE` environment variable.\n\n`Resource not accessible by integration` when posting the PR comment means the workflow token lacks write access. Make sure the job has `permissions: pull-requests: write`, as in the workflow above. PRs from forks get a read-only token by design; for public repos, expect the comment step to fail on fork PRs even though the eval itself runs.\n\n## Next steps\n\n- Add a second entry under `providers:` to compare models side by side in one results table. This is how you test a model upgrade before committing to it.\n- Set `fail-on-threshold: 90` on the action to allow a pass rate above 90% instead of requiring perfection, useful once your suite grows past a handful of cases.\n- Emit machine-readable results with `-o results.junit.xml` (also`.json` ,`.html` ) and feed them to your CI's test reporting.\n- Read promptfoo's [CI/CD integration guide](https://www.promptfoo.dev/docs/integrations/ci-cd/) for GitLab/Jenkins equivalents, and the[assertions reference](https://www.promptfoo.dev/docs/configuration/expected-outputs/) for the full assertion catalog, including semantic similarity and JSON schema validation.\n\n## Sources & further reading\n\n1. \n                                    [Getting started](https://www.promptfoo.dev/docs/getting-started/)\n                                — promptfoo.dev\n2. \n                                    [GitHub Actions integration](https://www.promptfoo.dev/docs/integrations/github-action/)\n                                — promptfoo.dev\n3. \n                                    [Anthropic provider](https://www.promptfoo.dev/docs/providers/anthropic/)\n                                — promptfoo.dev\n4. \n                                    [Command line reference](https://www.promptfoo.dev/docs/usage/command-line/)\n                                — promptfoo.dev\n5. \n                                    [Assertions and metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/)\n                                — promptfoo.dev\n6. \n                                    [promptfoo-action repository](https://github.com/promptfoo/promptfoo-action)\n                                — github.com\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor\n\nRachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/regression-test-your-prompts-with-promptfoo-in-ci", "canonical_source": "https://sourcefeed.dev/a/regression-test-your-prompts-with-promptfoo-in-ci", "published_at": "2026-09-07 11:41:58+00:00", "updated_at": "2026-09-07 11:55:22.527892+00:00", "lang": "en", "topics": ["developer-tools", "mlops", "artificial-intelligence"], "entities": ["promptfoo", "GitHub Actions", "Anthropic", "Claude", "Rachel Goldstein", "Node.js"], "alternates": {"html": "https://wpnews.pro/news/regression-test-your-prompts-with-promptfoo-in-ci", "markdown": "https://wpnews.pro/news/regression-test-your-prompts-with-promptfoo-in-ci.md", "text": "https://wpnews.pro/news/regression-test-your-prompts-with-promptfoo-in-ci.txt", "jsonld": "https://wpnews.pro/news/regression-test-your-prompts-with-promptfoo-in-ci.jsonld"}}