{"slug": "llm-regression-testing-in-ci-gate-pull-requests-on-eval-diffs", "title": "LLM regression testing in CI: gate pull requests on eval diffs", "summary": "EvalShift's GitHub Action gates pull requests on LLM eval diffs, failing CI when regressions exceed a configured threshold. The action runs a golden JSONL suite, pushes results to EvalShift Cloud, and compares against a base-branch baseline to report aggregate and per-slice pass-rate deltas. It requires an evalshift.yaml config, a committed suite, an EVALSHIFT_TOKEN with run:create and run:read scopes, and a model provider key, with costs roughly suite size × 2 models per run.", "body_md": "[← all posts](/blog)\n\n# LLM regression testing in CI: gate pull requests on eval diffs\n\nWire a golden suite into GitHub Actions so every pull request gets a paired eval run, a base-branch diff, and a check that fails on real regressions.\n\nAn eval suite you run when you remember to run it is not a gate. It is a habit, and habits fail exactly when the pressure is on — the Friday prompt tweak, the dependency bump that moves a model alias, the week everyone is shipping something else. The suite stays correct the entire time. Nobody runs it.\n\nThe version that changes outcomes is attached to the pull request. Someone edits a system prompt to fix one customer complaint, the paired run says tool selection dropped on the refund slice, the check goes red, and the branch does not merge until someone looks. The conversation on the PR is then about a measured delta rather than about whether the change felt risky. This post is how to wire that up with the EvalShift GitHub Action, and how to turn it on without halting your team's merges on day one.\n\n## ## What the gate has to answer\n\nA CI eval earns its runtime by answering three questions on every pull request, with no human in the loop:\n\n- +Did anything regress? The action pushes the completed run to EvalShift Cloud, asks the API for a\nbaseline run on the base branch, and reads\n`aggregate_delta.regressions`\n\noff the server-side diff. - +Where? The same diff carries\n`per_slice_deltas`\n\n— a`pass_rate_delta`\n\nper slice — so \"worse overall\" resolves to \"worse on the multilingual cases, flat everywhere else.\" - +Is it big enough to block? That one is not a measurement, it is a policy choice, and it is the only part of the gate you actually configure.\n\nThe first two are the same numbers you would read in the web app. The third is the `fail-on`\n\ninput,\nbelow.\n\n## ## Prerequisites\n\nAll four are hard requirements — the job fails without them:\n\n- +\n`evalshift.yaml`\n\ncommitted at the repository root, or the`config:`\n\ninput pointed at wherever it lives. Paths inside the config resolve relative to the config file's own directory, so a config in a subdirectory works unchanged. - +A golden JSONL suite committed, or the\n`suite:`\n\ninput pointed at it. The default is`golden.jsonl`\n\n. - +A repository or environment secret\n`EVALSHIFT_TOKEN`\n\nholding an`es_...`\n\nservice-account key scoped to`run:create`\n\nand`run:read`\n\n. Those two scopes are exactly what the action exercises:`run:create`\n\nfor`evalshift push`\n\n,`run:read`\n\nfor the baseline lookup and the diff fetch. - +A model provider key in the job env matching the models named in\n`evalshift.yaml`\n\n—`ANTHROPIC_API_KEY`\n\n,`OPENAI_API_KEY`\n\n, or`GEMINI_API_KEY`\n\n(`GOOGLE_API_KEY`\n\nworks too). A cross-provider migration needs both keys.\n\nEvery run makes real model calls and spends real credits. Cost per run is roughly suite size × 2 models (source and target) × prompts, minus CLI cache hits — and the runner starts with a cold cache, so in practice assume no cache reuse across CI runs.\n\nThat cost is the reason to scope the trigger. Narrowing `on.pull_request.paths`\n\nto your suite,\nprompts, and config keeps the gate off pull requests that cannot possibly move the numbers.\n\n## ## The workflow\n\n```\npermissions:\n  contents: read\n  pull-requests: write\n  issues: write\n  statuses: write\njobs:\n  evalshift:\n    runs-on: ubuntu-latest\n    env:\n      EVALSHIFT_NONINTERACTIVE: \"1\"\n      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n    steps:\n      - uses: actions/checkout@v7\n      - uses: babaliauskas/evalshift-action@v0\n        with:\n          token: ${{ secrets.EVALSHIFT_TOKEN }}\n          fail-on: regression\n```\n\nIt is a composite action, not a container, so what runs is a short ordered list you can reason\nabout: set up Python (3.14 by default), `pip install`\n\na pinned `evalshift`\n\nCLI version, then a\nstdlib-only helper script that runs the suite, pushes the completed run to EvalShift Cloud, asks\nthe API for a compatible baseline run on the base branch, fetches the server-side diff, keeps\nexactly one PR comment updated in place, and sets the `evalshift/regression`\n\ncommit status. All\nevaluation and statistics happen in the CLI; all diffing happens on the server. The action is the\nwrapper.\n\nTwo things about that list are worth knowing before you stare at a log. Install is not cached, so\nbudget roughly 20 to 60 seconds for it every run. And the helper captures command output instead of\nstreaming it, printing each command's stdout only after that command finishes — a long\n`evalshift all`\n\nlooks like a hung job right up until it isn't.\n\nOf the four permissions, only `contents: read`\n\nis strictly required; it is what `actions/checkout`\n\nneeds. The other three buy you feedback: `pull-requests: write`\n\nand `issues: write`\n\nfor the comment\n(PR comments are issue comments in the REST API), `statuses: write`\n\nfor the commit status. Drop\nthem and you get stderr warnings instead — the gate still fails the job correctly.\n\n## ## Choosing fail-on\n\n`fail-on` | fails when |\n|---|---|\n`never` | never — reports only |\n`regression` | `regression_count > 0` |\n`any-slice-regression` | any per-slice `pass_rate_delta < 0` |\n\nThe non-obvious part: `any-slice-regression`\n\nis not a superset of `regression`\n\n. They read different\nfields. A run with `regressions > 0`\n\nbut no negative slice delta fails under `regression`\n\nand passes\nunder `any-slice-regression`\n\n— so switching modes is a change of question, not a tightening of a\ndial.\n\nThe other case to have in your head is the empty one. When there is no comparable baseline diff —\nno run yet on the base branch, or the base branch resolves empty — the conclusion is success with\n`regression_count = 0`\n\n. The first pull request on a new project therefore never blocks, and neither\ndoes the first one after you rename a branch. A permanently green check usually means this, not\nthat your suite is passing.\n\n## ## Rolling it out without blocking everyone on day one\n\n- +Ship it with\n`fail-on: never`\n\nand leave it there for about a week. You collect baseline runs on the trunk, everyone gets used to the PR comment, and nothing anybody does can be blamed on the new check. - +Switch to\n`fail-on: regression`\n\nonce you have looked at a handful of comments and agree with what they said. This is the setting most teams should stay on. - +Add\n`any-slice-regression`\n\nonly when your slices are meaningful and each one carries enough paired observations to be testable at all — comparisons with fewer than 5 are skipped as insufficient, and a slice that keeps getting skipped will make this mode look erratic.\n\nDo not skip step one. The point of the calibration week is to find out whether your suite is noisy before that noise starts blocking merges, because a gate people learn to re-run until it passes is worse than no gate.\n\n## ## Gating locally too\n\nThe Cloud diff is not the only way to fail a build. The CLI gates on its own analysis, no Cloud layer involved:\n\n```\nevalshift all --gate critical,high\n```\n\n`--gate`\n\ntakes a comma-separated subset of `critical,high,medium,low`\n\nand exits 1 when any\ncomparison lands on a listed severity. `--policy-gate`\n\nexits 1 when the migration verdict is `fail`\n\nor `conditional_pass`\n\n. The two mechanisms answer different questions: `--gate`\n\nand `--policy-gate`\n\nlook at this run's own statistics, while the action's `fail-on`\n\nlooks at the diff against a\nbaseline. A regression that is already in your trunk shows up in the first and not the second.\n\nOne convenience worth knowing: when `$GITHUB_STEP_SUMMARY`\n\nis set, `analyze`\n\nappends a markdown\nresults table to it. The CLI inherits the job environment either way, so you get that summary on the\nrun page whether you invoke the CLI yourself or let the action do it.\n\n## ## Token hygiene\n\nThe token `evalshift login`\n\nissues is personal — it is tied to your membership and dies with it,\nwhich makes it exactly the wrong credential for a pipeline that has to outlive your employment.\nMint a service-account key instead, in the web app under Settings, API tokens, Service accounts,\nscope it to `run:create`\n\nand `run:read`\n\n, store it as an encrypted CI secret, and pass it as\n`EVALSHIFT_TOKEN`\n\n. Never run `login`\n\non a runner, and never expose the secret to\n`pull_request_target`\n\n, which runs the base repo's workflow with secrets in scope against fork code.\nRotation is overlapping keys, never an in-place swap: mint the successor, update the secret, confirm\na green run, then let the predecessor expire.\n\n## ## Keep reading\n\n- +\n[How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship)— the paired run this check automates. - +\n[When to trust an LLM judge](/blog/when-to-trust-an-llm-judge)— what pairwise judging is good at, and where it quietly misleads you. - +\n[Gating and PR feedback](/docs/action-gating)— every input, output, and failure mode of the action. - +\n[Cloud setup](/docs/hosted-setup)— projects, baselines, and the account side of the diff.", "url": "https://wpnews.pro/news/llm-regression-testing-in-ci-gate-pull-requests-on-eval-diffs", "canonical_source": "https://www.evalshift.dev/blog/llm-regression-testing-in-ci", "published_at": "2026-07-24 00:00:00+00:00", "updated_at": "2026-08-20 17:14:56.935106+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "mlops"], "entities": ["EvalShift", "GitHub Actions", "EvalShift Cloud", "babaliauskas/evalshift-action"], "alternates": {"html": "https://wpnews.pro/news/llm-regression-testing-in-ci-gate-pull-requests-on-eval-diffs", "markdown": "https://wpnews.pro/news/llm-regression-testing-in-ci-gate-pull-requests-on-eval-diffs.md", "text": "https://wpnews.pro/news/llm-regression-testing-in-ci-gate-pull-requests-on-eval-diffs.txt", "jsonld": "https://wpnews.pro/news/llm-regression-testing-in-ci-gate-pull-requests-on-eval-diffs.jsonld"}}