LLM regression testing in CI: gate pull requests on eval diffs

← all posts

Wire a golden suite into GitHub Actions so every pull request gets a paired eval run, a base-branch diff, and a check that fails on real regressions.

An eval suite you run when you remember to run it is not a gate. It is a habit, and habits fail exactly when the pressure is on — the Friday prompt tweak, the dependency bump that moves a model alias, the week everyone is shipping something else. The suite stays correct the entire time. Nobody runs it.

The version that changes outcomes is attached to the pull request. Someone edits a system prompt to fix one customer complaint, the paired run says tool selection dropped on the refund slice, the check goes red, and the branch does not merge until someone looks. The conversation on the PR is then about a measured delta rather than about whether the change felt risky. This post is how to wire that up with the EvalShift GitHub Action, and how to turn it on without halting your team's merges on day one.

## What the gate has to answer #

A CI eval earns its runtime by answering three questions on every pull request, with no human in the loop:

• +Did anything regress? The action pushes the completed run to EvalShift Cloud, asks the API for a baseline run on the base branch, and reads aggregate_delta.regressions

off the server-side diff. - +Where? The same diff carries per_slice_deltas

— apass_rate_delta

per slice — so "worse overall" resolves to "worse on the multilingual cases, flat everywhere else." - +Is it big enough to block? That one is not a measurement, it is a policy choice, and it is the only part of the gate you actually configure.

The first two are the same numbers you would read in the web app. The third is the fail-on

input, below.

## Prerequisites #

All four are hard requirements — the job fails without them:

evalshift.yaml

committed at the repository root, or theconfig:

input pointed at wherever it lives. Paths inside the config resolve relative to the config file's own directory, so a config in a subdirectory works unchanged. - +A golden JSONL suite committed, or the suite:

input pointed at it. The default isgolden.jsonl

. - +A repository or environment secret EVALSHIFT_TOKEN

holding anes_...

service-account key scoped torun:create

andrun:read

. Those two scopes are exactly what the action exercises:run:create

forevalshift push

,run:read

for the baseline lookup and the diff fetch. - +A model provider key in the job env matching the models named in evalshift.yaml

—ANTHROPIC_API_KEY

,OPENAI_API_KEY

, orGEMINI_API_KEY

(GOOGLE_API_KEY

works too). A cross-provider migration needs both keys.

Every run makes real model calls and spends real credits. Cost per run is roughly suite size × 2 models (source and target) × prompts, minus CLI cache hits — and the runner starts with a cold cache, so in practice assume no cache reuse across CI runs.

That cost is the reason to scope the trigger. Narrowing on.pull_request.paths

to your suite, prompts, and config keeps the gate off pull requests that cannot possibly move the numbers.

## The workflow #

permissions:
  contents: read
  pull-requests: write
  issues: write
  statuses: write
jobs:
  evalshift:
    runs-on: ubuntu-latest
    env:
      EVALSHIFT_NONINTERACTIVE: "1"
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    steps:
      - uses: actions/checkout@v7
      - uses: babaliauskas/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          fail-on: regression

It is a composite action, not a container, so what runs is a short ordered list you can reason about: set up Python (3.14 by default), pip install

a pinned evalshift

CLI version, then a stdlib-only helper script that runs the suite, pushes the completed run to EvalShift Cloud, asks the API for a compatible baseline run on the base branch, fetches the server-side diff, keeps exactly one PR comment updated in place, and sets the evalshift/regression

commit status. All evaluation and statistics happen in the CLI; all diffing happens on the server. The action is the wrapper.

Two things about that list are worth knowing before you stare at a log. Install is not cached, so budget roughly 20 to 60 seconds for it every run. And the helper captures command output instead of streaming it, printing each command's stdout only after that command finishes — a long evalshift all

looks like a hung job right up until it isn't.

Of the four permissions, only contents: read

is strictly required; it is what actions/checkout

needs. The other three buy you feedback: pull-requests: write

and issues: write

for the comment (PR comments are issue comments in the REST API), statuses: write

for the commit status. Drop them and you get stderr warnings instead — the gate still fails the job correctly.

## Choosing fail-on #

fail-on	fails when
never	never — reports only
regression	regression_count > 0
any-slice-regression	any per-slice pass_rate_delta < 0

The non-obvious part: any-slice-regression

is not a superset of regression

. They read different fields. A run with regressions > 0

but no negative slice delta fails under regression

and passes under any-slice-regression

— so switching modes is a change of question, not a tightening of a dial.

The other case to have in your head is the empty one. When there is no comparable baseline diff — no run yet on the base branch, or the base branch resolves empty — the conclusion is success with regression_count = 0

. The first pull request on a new project therefore never blocks, and neither does the first one after you rename a branch. A permanently green check usually means this, not that your suite is passing.

## Rolling it out without blocking everyone on day one #

• +Ship it with fail-on: never

and leave it there for about a week. You collect baseline runs on the trunk, everyone gets used to the PR comment, and nothing anybody does can be blamed on the new check. - +Switch to fail-on: regression

once you have looked at a handful of comments and agree with what they said. This is the setting most teams should stay on. - +Add any-slice-regression

only when your slices are meaningful and each one carries enough paired observations to be testable at all — comparisons with fewer than 5 are skipped as insufficient, and a slice that keeps getting skipped will make this mode look erratic.

Do not skip step one. The point of the calibration week is to find out whether your suite is noisy before that noise starts blocking merges, because a gate people learn to re-run until it passes is worse than no gate.

## Gating locally too #

The Cloud diff is not the only way to fail a build. The CLI gates on its own analysis, no Cloud layer involved:

evalshift all --gate critical,high

--gate

takes a comma-separated subset of critical,high,medium,low

and exits 1 when any comparison lands on a listed severity. --policy-gate

exits 1 when the migration verdict is fail

or conditional_pass

. The two mechanisms answer different questions: --gate

and --policy-gate

look at this run's own statistics, while the action's fail-on

looks at the diff against a baseline. A regression that is already in your trunk shows up in the first and not the second.

One convenience worth knowing: when $GITHUB_STEP_SUMMARY

is set, analyze

appends a markdown results table to it. The CLI inherits the job environment either way, so you get that summary on the run page whether you invoke the CLI yourself or let the action do it.

## Token hygiene #

The token evalshift login

issues is personal — it is tied to your membership and dies with it, which makes it exactly the wrong credential for a pipeline that has to outlive your employment. Mint a service-account key instead, in the web app under Settings, API tokens, Service accounts, scope it to run:create

and run:read

, store it as an encrypted CI secret, and pass it as EVALSHIFT_TOKEN

. Never run login

on a runner, and never expose the secret to pull_request_target

, which runs the base repo's workflow with secrets in scope against fork code. Rotation is overlapping keys, never an in-place swap: mint the successor, update the secret, confirm a green run, then let the predecessor expire.

## Keep reading #

How to test an LLM model migration before you ship it— the paired run this check automates. - + When to trust an LLM judge— what pairwise judging is good at, and where it quietly misleads you. - + Gating and PR feedback— every input, output, and failure mode of the action. - + Cloud setup— projects, baselines, and the account side of the diff.

source & further reading

evalshift.dev — original article What breaks when you switch LLMs? How to test an LLM model migration before you ship it