Evals in CI/CD and Regression Gates A lesson on evaluating agentic AI systems recommends wiring eval suites into CI/CD so that every change capable of altering agent behavior — agent code, system prompts, model versions, tool definitions, retrieval content, config, and the evals themselves — runs automatically and can block a merge. The lesson prescribes four tiers: a smoke suite of 20–30 cases with 1 trial under 3 minutes on every push, a regression suite of 150–300 cases plus a step suite with 2–3 trials under 20 minutes on every PR, a full nightly suite including adversarial splits with 3–5 trials, and a manual model-adoption run with 5 trials on new model versions. It argues teams should keep prompts, model names, tool schemas, and config as versioned repo files so each change is a diff CI can trigger on, and store a main-branch baseline report — per-case results, aggregate metrics, config fingerprint, dataset version, and judge model version — that is never compared across dataset versions. · Agentic AI · 9 min read 📋 Prerequisites - Online Evaluation and Production Monitoring previous lesson 🎯 What You'll Learn - Identify every kind of change that should trigger an agent evaluation - Design tiered eval suites that fit into PR, nightly, and release pipelines - Implement regression gates with baselines, hard gates, and statistically sound thresholds - Build a GitHub Actions workflow that runs evals and posts a report on the pull request The Eval Nobody Runs Most teams that build an eval framework eventually find it in the same state: it works, it’s useful, and someone has to remember to run it. So it runs before big launches, and in between, three prompt tweaks, a tool schema change, and a model version bump go out unmeasured. An eval only protects you if it runs automatically on every change that can affect behavior , and if its result can block that change. This lesson wires the framework into CI. What Counts as a Change For normal software, a change is a code diff. For an agent, behavior changes when any of these change: | Change | Example | Often missed? | |---|---|---| | Agent code | New loop logic, retry handling | No | | System prompt | “Be more concise” | Yes : often edited outside code review | | Model | Version bump, provider switch | Yes : sometimes changes without a deploy | | Tool definitions | Renamed parameter, new description | Yes | | Tool implementations | API now returns a different shape | Sometimes | | Retrieval / KB content | New help articles | Yes | | Config | Temperature, max steps | Sometimes | | The eval itself | Dataset, graders, judge rubric | Sometimes | The architectural fix from Lesson 2 pays off here: if prompts, model names, tool schemas, and config all live as versioned files in the repo , every one of these changes is a diff, and CI can trigger on it. If the system prompt lives in a database that product managers edit in a web UI, you’ve built an unmonitored path to production. Paths that should trigger the agent eval workflow on: pull request: paths: - "agent/ " - "prompts/ " - "tools/ " - "config/agent .yaml" - "evals/ " For changes that don’t arrive as diffs provider-side model updates, KB content changes , the nightly run against a pinned baseline catches them. Tiered Suites You can’t run 500 cases × 5 trials on every push. Split by when it runs this mirrors the dataset splits from Lesson 4 : | Tier | Trigger | Suite | Trials | Target time | Blocks | |---|---|---|---|---|---| | Smoke | Every push | 20–30 cases | 1 | under 3 min | Merge | | Regression | Every PR ready for review | 150–300 cases + step suite | 2–3 | under 20 min | Merge | | Full | Nightly, release candidates | All splits incl. adversarial | 3–5 | Hours OK | Release | | Model adoption | Manual, on new model version | Full + holdout | 5 | Hours OK | Model switch | Speed tricks that keep the PR tier tolerable: - Mock tool backends for everything except a handful of live smoke tests - High concurrency , bounded by provider rate limits - Cache judge verdicts keyed on evidence hash Lesson 6 - Run judges only on runs that passed gates - Selective re-running : if only the KB changed, run the cases tagged uses:kb first Baselines A gate needs something to compare against. Store a baseline report for the main branch: per-case results, aggregate metrics, the agent config fingerprint, dataset version, and judge model version. php def load baseline path="evals/baselines/main.json" - dict: with open path as f: return json.load f Refresh the baseline when a change merges to main. Store it as an artifact, in object storage, or in your eval platform. Don’t compare across dataset versions. If a PR changes the dataset, rerun the baseline config on the new dataset in the same CI job. Otherwise you’re comparing scores on different exams. Gate Design A good gate has three layers, from strictest to most nuanced. 1. Hard Gates: Zero Tolerance Any failure blocks the merge, regardless of averages: - Any policy gate failure on any case refund limit, cross-account, leaks - Any adversarial case that passed on baseline and now fails - Any case tagged critical that regresses - Smoke-suite pass rate below 100%, if the smoke suite is truly smoke 2. Statistical Regression Gate For the aggregate quality score, use the paired comparison from Lesson 10. Block only when there’s evidence of a regression, not when the score wobbles. php def regression gate baseline: dict, candidate: dict, max drop: float = 0.02 - dict: """Fail if we're confident the pass rate dropped by more than max drop.""" cmp = paired bootstrap baseline "per case" , candidate "per case" lo, hi = cmp "ci95" if hi < -max drop: verdict = "fail" confidently worse than tolerance elif lo < -max drop: verdict = "warn" can't rule out a meaningful drop else: verdict = "pass" return { cmp, "verdict": verdict} The three-way verdict matters. “Warn” means “the suite can’t tell”. The author should look at regressed cases, rerun with more trials, or accept the risk explicitly. Treating every dip as a failure makes the gate flaky and trains people to override it. Treating no dip as a failure lets slow erosion through. 3. Slice Gates Aggregate scores hide localized damage. Check key slices separately: CRITICAL SLICES = {"risk:refund under limit": 0.03, "risk:refund over limit": 0.0, "intent:cancellation": 0.05} def slice gates baseline, candidate - list str : failures = for tag, tolerance in CRITICAL SLICES.items : b, c = baseline "by tag" .get tag , candidate "by tag" .get tag if b is not None and c is not None and c < b - tolerance: failures.append f"{tag}: {b:.1%} → {c:.1%} tolerance {tolerance:.0%} " return failures 4. Budget Gates Quality isn’t the only thing that regresses: - Mean cost per task up more than 25% - p95 latency up more than 30% - Mean tool calls up more than 20% These usually warn rather than block. Sometimes a cost increase is a deliberate tradeoff, and the author should say so in the PR. The Eval Runner Script One entry point that CI calls, which runs a tier and writes a machine-readable result and a human-readable report: python evals/run.py import argparse, asyncio, json, sys def main : ap = argparse.ArgumentParser ap.add argument "--tier", choices= "smoke", "regression", "full" , required=True ap.add argument "--trials", type=int, default=1 ap.add argument "--baseline", default="evals/baselines/main.json" ap.add argument "--out", default="eval-results" args = ap.parse args config = load agent config "config/agent.yaml" cases = load cases "evals/datasets/quill.jsonl", splits=SPLITS args.tier runs = asyncio.run Runner make quill, config, trials=args.trials .run cases report = build report cases, runs, QUILL SCORECARD baseline = load baseline args.baseline gates = { "hard": hard gate failures report, baseline , "regression": regression gate baseline, report , "slices": slice gates baseline, report , "budget": budget warnings baseline, report , } blocked = bool gates "hard" or gates "regression" "verdict" == "fail" \ or bool gates "slices" write json f"{args.out}/report.json", {"report": report, "gates": gates} write text f"{args.out}/summary.md", render markdown report, baseline, gates sys.exit 1 if blocked else 0 if name == " main ": main The GitHub Actions Workflow name: agent-evals on: pull request: paths: "agent/ ", "prompts/ ", "tools/ ", "config/agent .yaml", "evals/ " schedule: - cron: "0 6 " nightly full run workflow dispatch: inputs: tier: { description: "smoke | regression | full", default: "full" } concurrency: group: evals-${{ github.ref }} cancel-in-progress: true jobs: smoke: if: github.event name == 'pull request' runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.12", cache: "pip" } - run: pip install -r evals/requirements.txt - run: python evals/run.py --tier smoke --trials 1 env: LLM API KEY: ${{ secrets.LLM API KEY }} regression: needs: smoke if: github.event name == 'pull request' && github.event.pull request.draft runs-on: ubuntu-latest timeout-minutes: 45 steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.12", cache: "pip" } - run: pip install -r evals/requirements.txt - name: Fetch main baseline run: gh run download --name eval-baseline-main --dir evals/baselines || true env: { GH TOKEN: "${{ github.token }}" } - name: Run regression suite id: evals run: python evals/run.py --tier regression --trials 3 env: LLM API KEY: ${{ secrets.LLM API KEY }} - name: Post report to PR if: always uses: marocchino/sticky-pull-request-comment@v2 with: header: agent-evals path: eval-results/summary.md - uses: actions/upload-artifact@v4 if: always with: { name: eval-results-${{ github.sha }}, path: eval-results/ } nightly: if: github.event name == 'schedule' || github.event name == 'workflow dispatch' runs-on: ubuntu-latest timeout-minutes: 240 steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.12", cache: "pip" } - run: pip install -r evals/requirements.txt - run: python evals/run.py --tier ${{ inputs.tier || 'full' }} --trials 5 env: LLM API KEY: ${{ secrets.LLM API KEY }} - uses: actions/upload-artifact@v4 if: always with: { name: eval-nightly-${{ github.run id }}, path: eval-results/ } A separate small workflow on push to main reruns the regression tier and uploads the result as the eval-baseline-main artifact. Mark the regression job as a required status check in branch protection. Without that, the gate is advisory. The PR Report The Markdown summary is what reviewers actually read. Make it scannable, lead with the verdict, and link to traces: 🧪 Agent eval — regression tier 240 cases × 3 trials Verdict: ⚠️ WARN — can't rule out a drop larger than 2 pts | Metric | main | this PR | Δ | |---|---|---|---| | Pass rate | 86.3% | 84.9% | −1.4 pts 95% CI −3.6, +0.8 | | pass^3 | 71.2% | 68.8% | −2.4 pts | | Refund slice | 91.0% | 91.7% | +0.7 | | Cost / task | $0.021 | $0.019 | −9% | | p95 latency | 11.8s | 10.9s | −8% | Hard gates: ✅ none failed Regressed cases 7 : dup-0031 , plan-0012 , cancel-0004 … traces Improved cases 4 : refund-0019 , … traces Handling Flakiness and Overrides Flaky gates destroy trust faster than missing gates. If a gate fails, someone reruns, and it passes, people learn the gate is noise. Reduce flakiness with more trials in the PR tier, the three-way verdict, temperature 0 where the agent allows it, and mocked tools. Overrides should be explicit and visible. Sometimes a regression is acceptable: a deliberate tradeoff, or a known-bad case being redefined. Allow an override via a PR label eval-override that requires a written justification in the PR description and a review from an eval owner. Log every override. If overrides become routine, the gate thresholds are wrong. Changing the eval is a change. A PR that edits expectations, graders, or judge rubrics should be reviewed by someone who owns the eval, and should show the before/after score on the old agent so reviewers can see that the eval changed, not the agent. Model Upgrades as Pull Requests Treat a model version change exactly like code: a PR that changes config/agent.yaml from one model ID to another. It runs the regression tier automatically, then the model-adoption tier manually. The PR report becomes the record of why you did, or didn’t, adopt the new model. Pin model versions explicitly. A floating alias that the provider updates silently bypasses all of this. Key Takeaways - Prompts, models, tool schemas, KB content, and config all change behavior. Make them versioned files so CI can see them. - Run tiered suites : smoke on every push, regression on every PR, full nightly, model adoption on demand - Gate in layers: hard gates zero tolerance , a statistical regression gate with a pass/warn/fail verdict, slice gates , and budget warnings - Compare against a baseline on the same dataset version - Make the regression job a required check ; make overrides explicit, justified, and logged - Treat model upgrades and eval changes as reviewed pull requests Exercise 1. Move your agent’s prompt, model ID, tool schemas, and config into versioned files 2. Implement evals/run.py with smoke and regression tiers and the four gate layers 3. Add the GitHub Actions workflow and the baseline-refresh workflow 4. Open a PR that deliberately breaks something subtle for example, remove “look up the invoice before refunding” from the prompt and confirm the gate catches it and the report points to the right cases