Give Every Agent Patch a Regression Budget, Not Just a Green Check

A green CI run is not a verdict. It is a photograph taken at one moment, under one seed, with one set of fixtures. When the photographer is an agent that rewrites five files in a single pass, you need more than a photograph — you need a budget.

This article shows how to define a regression budget for agent-generated patches: a fixed number of properties, fixture contracts, and flaky-test quarantine slots that the patch must respect before it merges. I use free model access and a free server tier to keep the gate honest, but the budgeting idea works with any CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A checklist says: run property P, run fixture F, mark flaky test T. A budget says: here is how much verification you can afford, here is what must pass, and here is what gets cut when time runs out.

Budgeting forces prioritization. Agent patches fail in noisy ways: an LLM can pass eight property checks and then break a fixture contract that was never loaded. If you have a fixed number of verification credits, you will naturally put the highest-signal checks first.

I split regression verification into three buckets. Each bucket has a cost, measured in free-tier compute minutes, and a cap.

This table is small on purpose. A budget you cannot explain in one table is a budget your CI will ignore.

I used a shell script that reads a small YAML file describing the budget. The script runs three phases and stops as soon as one phase exceeds its cap.

#!/usr/bin/env bash
set -euo pipefail

BUDGET_FILE="budget.yml"
phase_cap() {
  local phase=$1
  grep -A4 "$phase:" "$BUDGET_FILE" | grep cap | awk '{print $2}'
}

run_verify() {
  local phase=$1
  local cap=$2
  local start=$(date +%s)
  ./verify_$phase.sh || true
  local elapsed=$(( $(date +%s) - start ))
  if (( elapsed > cap * 60 )); then
    echo "Budget exceeded for $phase"
    exit 1
  fi
}

run_verify properties $(phase_cap properties)
run_verify fixtures   $(phase_cap fixtures)
run_verify quarantine $(phase_cap quarantine)

This is pseudocode, but it is runnable if you have verify_properties.sh

, verify_fixtures.sh

, and verify_quarantine.sh

lying around. I deliberately label it as a scaffold, not a production tool.

The most useful artifact I have built is a fixture inventory that records what each fixture locks and when it was last verified. It turns "run all tests" into "verify the contract the patch actually touched."

user_service:
  version: "2026-09-01"
  locks:
    - auth.token_expiry
    - rate_limit.retry_after
    - user.update.immutable_fields
  verified_at: "2026-08-31T10:00:00Z"
  agent_patch_impact: "changed"

payment_splitter:
  version: "2026-08-15"
  locks:
    - split.amount_precision
    - split.currency_default
  verified_at: null
  agent_patch_impact: "untouched"

Before an agent patch is allowed through, the gate compares the list of files changed against this inventory. Only fixtures with agent_patch_impact: changed

run in the fixture bucket. The others sit until their contract is actually at risk.

This is not an optimization. It is an honesty check: if a patch touches auth.py

, you do not need to run the payment_splitter suite to judge it.

Free model access and a free server tier change two things: the cost of generating a patch and the cost of verifying it. When both are near zero, the temptation is to run more. Budgeting says: run less, but run the right thing.

I ran this budget gate on a free server instance with a two-core limit. The property bucket consumed about 12 minutes for 200 cases. The fixture bucket consumed 6 minutes for 14 changed fixtures. The quarantine bucket took 2 minutes because no flaky test failed twice.

Total: 20 minutes per patch. That is long enough to matter, short enough to keep an agent loop human-supervised.

This budget model assumes your tests are deterministic enough that a quarantine list is small. If your suite is 40% flaky, no budget will fix that; you need to fix the flakiness first.

The script above is a scaffold, not a scalable tool. It does not handle parallel CI workers, distributed caches, or multi-service fixture startup. If your system needs those, use a purpose-built platform and keep the budget idea, not the script.

You should not use this approach when your patch changes a security-critical module, where a narrow budget is dangerous. Security changes deserve exhaustive verification, not a cost cap.

A regression budget is not about saving money. It is about forcing the gate to decide what cannot be skipped — before the agent's merge button gets pressed.

If you want to see how this budget behaves against 40 injected bugs, the same fixture inventory can be replayed with real failures. I recommend doing that once per month, with a deliberately broken patch, to check that the gate still cares.