cd /news/ai-agents/evals-in-ci-cd-and-regression-gates · home › topics › ai-agents › article
[ARTICLE · art-172291] src=superml.org ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Evals in CI/CD and Regression Gates

A lesson on evaluating agentic AI systems recommends wiring eval suites into CI/CD so that every change capable of altering agent behavior — agent code, system prompts, model versions, tool definitions, retrieval content, config, and the evals themselves — runs automatically and can block a merge. The lesson prescribes four tiers: a smoke suite of 20–30 cases with 1 trial under 3 minutes on every push, a regression suite of 150–300 cases plus a step suite with 2–3 trials under 20 minutes on every PR, a full nightly suite including adversarial splits with 3–5 trials, and a manual model-adoption run with 5 trials on new model versions. It argues teams should keep prompts, model names, tool schemas, and config as versioned repo files so each change is a diff CI can trigger on, and store a main-branch baseline report — per-case results, aggregate metrics, config fingerprint, dataset version, and judge model version — that is never compared across dataset versions.

recommendation

Split agent evals into a sub-3-minute smoke tier on every push and a sub-20-minute regression tier on every PR, and rerun the baseline config on any new dataset version before comparing scores.

more in playbook: Evals and testing for LLM apps
read10 min views2 publishedNov 18, 2026
Evals in CI/CD and Regression Gates
Image: Superml (auto-discovered)

· Agentic AI · 9 min read

📋 Prerequisites

  • Online Evaluation and Production Monitoring (previous lesson)

🎯 What You'll Learn

  • Identify every kind of change that should trigger an agent evaluation
  • Design tiered eval suites that fit into PR, nightly, and release pipelines
  • Implement regression gates with baselines, hard gates, and statistically sound thresholds
  • Build a GitHub Actions workflow that runs evals and posts a report on the pull request

The Eval Nobody Runs #

Most teams that build an eval framework eventually find it in the same state: it works, it’s useful, and someone has to remember to run it. So it runs before big launches, and in between, three prompt tweaks, a tool schema change, and a model version bump go out unmeasured.

An eval only protects you if it runs automatically on every change that can affect behavior, and if its result can block that change. This lesson wires the framework into CI.

What Counts as a Change #

For normal software, a change is a code diff. For an agent, behavior changes when any of these change:

Change Example Often missed?
Agent code New loop logic, retry handling No
System prompt “Be more concise” Yes : often edited outside code review
Model Version bump, provider switch Yes : sometimes changes without a deploy
Tool definitions Renamed parameter, new description Yes
Tool implementations API now returns a different shape Sometimes
Retrieval / KB content New help articles Yes
Config Temperature, max steps Sometimes
The eval itself Dataset, graders, judge rubric Sometimes

The architectural fix from Lesson 2 pays off here: if prompts, model names, tool schemas, and config all live as versioned files in the repo, every one of these changes is a diff, and CI can trigger on it. If the system prompt lives in a database that product managers edit in a web UI, you’ve built an unmonitored path to production.

on:
  pull_request:
    paths:
      - "agent/**"
      - "prompts/**"
      - "tools/**"
      - "config/agent*.yaml"
      - "evals/**"

For changes that don’t arrive as diffs (provider-side model updates, KB content changes), the nightly run against a pinned baseline catches them.

Tiered Suites #

You can’t run 500 cases × 5 trials on every push. Split by when it runs (this mirrors the dataset splits from Lesson 4):

Tier Trigger Suite Trials Target time Blocks
Smoke Every push 20–30 cases 1 under 3 min Merge
Regression Every PR (ready for review) 150–300 cases + step suite 2–3 under 20 min Merge
Full Nightly, release candidates All splits incl. adversarial 3–5 Hours OK Release
Model adoption Manual, on new model version Full + holdout 5 Hours OK Model switch

Speed tricks that keep the PR tier tolerable:

  • Mock tool backends for everything except a handful of live smoke tests
  • High concurrency , bounded by provider rate limits
  • Cache judge verdicts keyed on evidence hash (Lesson 6)
  • Run judges only on runs that passed gates
  • Selective re-running : if only the KB changed, run the cases taggeduses:kb first

Baselines #

A gate needs something to compare against. Store a baseline report for the main branch: per-case results, aggregate metrics, the agent config fingerprint, dataset version, and judge model version.

def load_baseline(path="evals/baselines/main.json") -> dict:
    with open(path) as f:
        return json.load(f)

Refresh the baseline when a change merges to main. Store it as an artifact, in object storage, or in your eval platform.

Don’t compare across dataset versions. If a PR changes the dataset, rerun the baseline config on the new dataset in the same CI job. Otherwise you’re comparing scores on different exams.

Gate Design #

A good gate has three layers, from strictest to most nuanced.

1. Hard Gates: Zero Tolerance

Any failure blocks the merge, regardless of averages:

  • Any policy gate failure on any case (refund limit, cross-account, leaks)
  • Any adversarial case that passed on baseline and now fails
  • Any case tagged critical that regresses
  • Smoke-suite pass rate below 100%, if the smoke suite is truly smoke

2. Statistical Regression Gate

For the aggregate quality score, use the paired comparison from Lesson 10. Block only when there’s evidence of a regression, not when the score wobbles.

def regression_gate(baseline: dict, candidate: dict, max_drop: float = 0.02) -> dict:
    """Fail if we're confident the pass rate dropped by more than max_drop."""
    cmp = paired_bootstrap(baseline["per_case"], candidate["per_case"])
    lo, hi = cmp["ci95"]
    if hi < -max_drop:
        verdict = "fail"            # confidently worse than tolerance
    elif lo < -max_drop:
        verdict = "warn"            # can't rule out a meaningful drop
    else:
        verdict = "pass"
    return {**cmp, "verdict": verdict}

The three-way verdict matters. “Warn” means “the suite can’t tell”. The author should look at regressed cases, rerun with more trials, or accept the risk explicitly. Treating every dip as a failure makes the gate flaky and trains people to override it. Treating no dip as a failure lets slow erosion through.

3. Slice Gates

Aggregate scores hide localized damage. Check key slices separately:

CRITICAL_SLICES = {"risk:refund_under_limit": 0.03, "risk:refund_over_limit": 0.0,
                   "intent:cancellation": 0.05}

def slice_gates(baseline, candidate) -> list[str]:
    failures = []
    for tag, tolerance in CRITICAL_SLICES.items():
        b, c = baseline["by_tag"].get(tag), candidate["by_tag"].get(tag)
        if b is not None and c is not None and c < b - tolerance:
            failures.append(f"{tag}: {b:.1%} → {c:.1%} (tolerance {tolerance:.0%})")
    return failures

4. Budget Gates

Quality isn’t the only thing that regresses:

  • Mean cost per task up more than 25%
  • p95 latency up more than 30%
  • Mean tool calls up more than 20%

These usually warn rather than block. Sometimes a cost increase is a deliberate tradeoff, and the author should say so in the PR.

The Eval Runner Script #

One entry point that CI calls, which runs a tier and writes a machine-readable result and a human-readable report:

import argparse, asyncio, json, sys

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--tier", choices=["smoke", "regression", "full"], required=True)
    ap.add_argument("--trials", type=int, default=1)
    ap.add_argument("--baseline", default="evals/baselines/main.json")
    ap.add_argument("--out", default="eval-results")
    args = ap.parse_args()

    config = load_agent_config("config/agent.yaml")
    cases = load_cases("evals/datasets/quill.jsonl", splits=SPLITS[args.tier])
    runs = asyncio.run(Runner(make_quill, config, trials=args.trials).run(cases))
    report = build_report(cases, runs, QUILL_SCORECARD)

    baseline = load_baseline(args.baseline)
    gates = {
        "hard": hard_gate_failures(report, baseline),
        "regression": regression_gate(baseline, report),
        "slices": slice_gates(baseline, report),
        "budget": budget_warnings(baseline, report),
    }
    blocked = bool(gates["hard"]) or gates["regression"]["verdict"] == "fail" \
              or bool(gates["slices"])

    write_json(f"{args.out}/report.json", {"report": report, "gates": gates})
    write_text(f"{args.out}/summary.md", render_markdown(report, baseline, gates))
    sys.exit(1 if blocked else 0)

if __name__ == "__main__":
    main()

The GitHub Actions Workflow #

name: agent-evals

on:
  pull_request:
    paths: ["agent/**", "prompts/**", "tools/**", "config/agent*.yaml", "evals/**"]
  schedule:
    - cron: "0 6 * * *"          # nightly full run
  workflow_dispatch:
    inputs:
      tier: { description: "smoke | regression | full", default: "full" }

concurrency:
  group: evals-${{ github.ref }}
  cancel-in-progress: true

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: "pip" }
      - run: pip install -r evals/requirements.txt
      - run: python evals/run.py --tier smoke --trials 1
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

  regression:
    needs: smoke
    if: github.event_name == 'pull_request' && !github.event.pull_request.draft
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: "pip" }
      - run: pip install -r evals/requirements.txt
      - name: Fetch main baseline
        run: gh run download --name eval-baseline-main --dir evals/baselines || true
        env: { GH_TOKEN: "${{ github.token }}" }
      - name: Run regression suite
        id: evals
        run: python evals/run.py --tier regression --trials 3
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      - name: Post report to PR
        if: always()
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: agent-evals
          path: eval-results/summary.md
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-results-${{ github.sha }}, path: eval-results/ }

  nightly:
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    timeout-minutes: 240
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: "pip" }
      - run: pip install -r evals/requirements.txt
      - run: python evals/run.py --tier ${{ inputs.tier || 'full' }} --trials 5
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-nightly-${{ github.run_id }}, path: eval-results/ }

A separate small workflow on push to main reruns the regression tier and uploads the result as the eval-baseline-main artifact.

Mark the regression job as a required status check in branch protection. Without that, the gate is advisory.

The PR Report #

The Markdown summary is what reviewers actually read. Make it scannable, lead with the verdict, and link to traces:

## 🧪 Agent eval — regression tier (240 cases × 3 trials)

**Verdict: ⚠️ WARN** — can't rule out a drop larger than 2 pts

| Metric | main | this PR | Δ |
|---|---|---|---|
| Pass rate | 86.3% | 84.9% | −1.4 pts (95% CI −3.6, +0.8) |
| pass^3 | 71.2% | 68.8% | −2.4 pts |
| Refund slice | 91.0% | 91.7% | +0.7 |
| Cost / task | $0.021 | $0.019 | −9% |
| p95 latency | 11.8s | 10.9s | −8% |

**Hard gates:** ✅ none failed
**Regressed cases (7):** `dup-0031`, `plan-0012`, `cancel-0004` … [traces](#)
**Improved cases (4):** `refund-0019`, … [traces](#)

Handling Flakiness and Overrides #

Flaky gates destroy trust faster than missing gates. If a gate fails, someone reruns, and it passes, people learn the gate is noise. Reduce flakiness with more trials in the PR tier, the three-way verdict, temperature 0 where the agent allows it, and mocked tools.

Overrides should be explicit and visible. Sometimes a regression is acceptable: a deliberate tradeoff, or a known-bad case being redefined. Allow an override via a PR label (eval-override) that requires a written justification in the PR description and a review from an eval owner. Log every override. If overrides become routine, the gate thresholds are wrong.

Changing the eval is a change. A PR that edits expectations, graders, or judge rubrics should be reviewed by someone who owns the eval, and should show the before/after score on the old agent so reviewers can see that the eval changed, not the agent.

Model Upgrades as Pull Requests #

Treat a model version change exactly like code: a PR that changes config/agent.yaml from one model ID to another. It runs the regression tier automatically, then the model-adoption tier manually. The PR report becomes the record of why you did, or didn’t, adopt the new model. Pin model versions explicitly. A floating alias that the provider updates silently bypasses all of this.

Key Takeaways #

  • Prompts, models, tool schemas, KB content, and config all change behavior. Make them versioned files so CI can see them.
  • Run tiered suites : smoke on every push, regression on every PR, full nightly, model adoption on demand
  • Gate in layers: hard gates (zero tolerance), astatistical regression gate with a pass/warn/fail verdict,slice gates , andbudget warnings
  • Compare against a baseline on the same dataset version
  • Make the regression job a required check ; make overrides explicit, justified, and logged
  • Treat model upgrades and eval changes as reviewed pull requests

Exercise #

  1. Move your agent’s prompt, model ID, tool schemas, and config into versioned files
  2. Implement evals/run.py with smoke and regression tiers and the four gate layers
  3. Add the GitHub Actions workflow and the baseline-refresh workflow
  4. Open a PR that deliberately breaks something subtle (for example, remove “look up the invoice before refunding” from the prompt) and confirm the gate catches it and the report points to the right cases
── more in #ai-agents 4 stories · sorted by recency
── more on @github actions 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/evals-in-ci-cd-and-r…] indexed:0 read:10min 2026-11-18 · —