Give Every Agent Patch a Regression Budget, Not Just a Green Check

MonkeyCode's developer proposes a regression budget for agent-generated patches, replacing a simple green CI check with a fixed allocation of property tests, fixture contracts, and flaky-test quarantine slots. The approach, demonstrated with a shell script and YAML configuration, prioritizes high-signal verification and runs on free-tier compute to keep costs near zero.

A green CI run is not a verdict. It is a photograph taken at one moment, under one seed, with one set of fixtures. When the photographer is an agent that rewrites five files in a single pass, you need more than a photograph — you need a budget. This article shows how to define a regression budget for agent-generated patches: a fixed number of properties, fixture contracts, and flaky-test quarantine slots that the patch must respect before it merges. I use free model access and a free server tier to keep the gate honest, but the budgeting idea works with any CI. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A checklist says: run property P, run fixture F, mark flaky test T. A budget says: here is how much verification you can afford, here is what must pass, and here is what gets cut when time runs out. Budgeting forces prioritization. Agent patches fail in noisy ways: an LLM can pass eight property checks and then break a fixture contract that was never loaded. If you have a fixed number of verification credits, you will naturally put the highest-signal checks first. I split regression verification into three buckets. Each bucket has a cost, measured in free-tier compute minutes, and a cap. | Bucket | Artifact | Cap free tier | Pass condition | |---|---|---|---| | Properties | Hypothesis-based property tests | 200 cases per property | No counterexample found | | Fixtures | Versioned contract fixtures | 20 fixtures per patch | All fixture checks pass | | Quarantine | Flaky-test freeze list | 5 entries | No frozen test failed twice | This table is small on purpose. A budget you cannot explain in one table is a budget your CI will ignore. I used a shell script that reads a small YAML file describing the budget. The script runs three phases and stops as soon as one phase exceeds its cap. bash /usr/bin/env bash set -euo pipefail BUDGET FILE="budget.yml" phase cap { local phase=$1 grep -A4 "$phase:" "$BUDGET FILE" | grep cap | awk '{print $2}' } run verify { local phase=$1 local cap=$2 local start=$ date +%s ./verify $phase.sh || true local elapsed=$ $ date +%s - start if elapsed cap 60 ; then echo "Budget exceeded for $phase" exit 1 fi } run verify properties $ phase cap properties run verify fixtures $ phase cap fixtures run verify quarantine $ phase cap quarantine This is pseudocode, but it is runnable if you have verify properties.sh , verify fixtures.sh , and verify quarantine.sh lying around. I deliberately label it as a scaffold, not a production tool. The most useful artifact I have built is a fixture inventory that records what each fixture locks and when it was last verified. It turns "run all tests" into "verify the contract the patch actually touched." fixture inventory.yml user service: version: "2026-09-01" locks: - auth.token expiry - rate limit.retry after - user.update.immutable fields verified at: "2026-08-31T10:00:00Z" agent patch impact: "changed" payment splitter: version: "2026-08-15" locks: - split.amount precision - split.currency default verified at: null agent patch impact: "untouched" Before an agent patch is allowed through, the gate compares the list of files changed against this inventory. Only fixtures with agent patch impact: changed run in the fixture bucket. The others sit until their contract is actually at risk. This is not an optimization. It is an honesty check: if a patch touches auth.py , you do not need to run the payment splitter suite to judge it. Free model access and a free server tier change two things: the cost of generating a patch and the cost of verifying it. When both are near zero, the temptation is to run more. Budgeting says: run less, but run the right thing. I ran this budget gate on a free server instance with a two-core limit. The property bucket consumed about 12 minutes for 200 cases. The fixture bucket consumed 6 minutes for 14 changed fixtures. The quarantine bucket took 2 minutes because no flaky test failed twice. Total: 20 minutes per patch. That is long enough to matter, short enough to keep an agent loop human-supervised. This budget model assumes your tests are deterministic enough that a quarantine list is small. If your suite is 40% flaky, no budget will fix that; you need to fix the flakiness first. The script above is a scaffold, not a scalable tool. It does not handle parallel CI workers, distributed caches, or multi-service fixture startup. If your system needs those, use a purpose-built platform and keep the budget idea, not the script. You should not use this approach when your patch changes a security-critical module, where a narrow budget is dangerous. Security changes deserve exhaustive verification, not a cost cap. A regression budget is not about saving money. It is about forcing the gate to decide what cannot be skipped — before the agent's merge button gets pressed. If you want to see how this budget behaves against 40 injected bugs, the same fixture inventory can be replayed with real failures. I recommend doing that once per month, with a deliberately broken patch, to check that the gate still cares.