Benchmark Budgets Are Part of the Grader: A Paired Cost and Latency Gate for AI C++ Patches

A developer published a paired cost-and-latency gate script, budget_gate.py, that sits after correctness tests in AI-generated C++ patch evaluations and flags when a candidate patch set becomes more expensive per accepted patch. The tool reads baseline and candidate JSONL files sharing a suite hash and case IDs, takes medians over repeats, requires every repeat to pass, and gates the p95 wall and compile ratios, median token ratio, and p90 per-case token ratio, with queue wait reported separately as a warning.

Consider a pull request that turns a 12-line C++ change into a 40-line patch. Every golden case passes. CI is green. Two days later the review queue tells a different story: the patch needed three retries per case, pushed p95 compile time up by a third, and burned a quarter more tokens than the model it replaced. The grader never noticed, because it only asked whether the tests passed. That gap is the subject of this article. The latest benchmarking challenge wrap-up https://dev.to/devteam/congrats-to-the-kaggle-benchmarking-challenge-winners-2moh is a useful reminder that leaderboards are snapshots, not budgets. For AI-generated C++ patches, a passing score has a second dimension: tokens, compile time, and wall-clock time per accepted patch. A correctness-only harness treats every passing patch as equal. A patch that compiles in 200 ms and one that compiles in 2 s both get a check mark. A patch that uses 800 tokens and one that uses 8,000 tokens both count as one success. That is fine for a smoke test. It is not fine for a merge gate. Token allowances are finite, CI minutes are finite, and reviewer attention is finite. When a model swap improves pass rate by two points but doubles the tail latency, the eval has found a trade-off; the harness should report it instead of hiding it. The gate below is deliberately small. It does not replace correctness tests, prompt lock files, or flaky-case triage. It sits after them and answers one narrow question: did the candidate patch set get more expensive per accepted patch? The gate reads two JSONL files, one for the baseline and one for the candidate. Both must share a suite hash and the same case IDs. For each case, the script takes the median over repeats, requires every repeat to pass, and computes candidate-to-baseline ratios. It gates the 95th percentile of per-case wall and compile ratios, the median token ratio, and the 90th percentile of per-case token ratios. Queue wait is reported separately as a warning because shared capacity is noisy. bash /usr/bin/env python3 '''budget gate.py: paired cost and latency gate for AI C++ patch evals.''' import argparse import json import sys from collections import defaultdict from pathlib import Path from statistics import median REQUIRED = { 'run id', 'case id', 'variant', 'suite hash', 'server class', 'passed', 'prompt tokens', 'completion tokens', 'compile ms', 'exec ms', 'queue wait ms', } LIMITS = { 'p95 wall ratio': 1.20, 'p95 compile ratio': 1.25, 'median tokens ratio': 1.15, 'p90 case tokens ratio': 1.50, 'queue wait p95 ms': 30 000, 'min paired cases': 20, } def pct values, q : values = sorted values if not values: return 0.0 k = len values - 1 q lo = int k hi = min lo + 1, len values - 1 return values lo + values hi - values lo k - lo def load path : rows = for n, line in enumerate Path path .read text .splitlines , 1 : if not line.strip : continue try: row = json.loads line except json.JSONDecodeError as exc: raise SystemExit f'{path}:{n}: {exc}' missing = REQUIRED - set row if missing: raise SystemExit f'{path}:{n}: missing {sorted missing }' rows.append row return rows def by case rows, variant : grouped = defaultdict list for row in rows: if row 'variant' == variant: grouped row 'case id' .append row return grouped def case stats rows : return { 'passed': all bool row 'passed' for row in rows , 'wall': median float row 'compile ms' + float row 'exec ms' for row in rows , 'compile': median float row 'compile ms' for row in rows , 'tokens': median float row 'prompt tokens' + float row 'completion tokens' for row in rows , 'queue': median float row 'queue wait ms' for row in rows , } def ratio candidate, baseline : return candidate / baseline if baseline 0 else float 'inf' def main : parser = argparse.ArgumentParser parser.add argument '--baseline', required=True parser.add argument '--candidate', required=True parser.add argument '--report', default='budget-report.json' args = parser.parse args base = load args.baseline cand = load args.candidate for name, rows in 'baseline', base , 'candidate', cand : hashes = {row 'suite hash' for row in rows} if len hashes = 1: raise SystemExit f'{name}: mixed suite hash {hashes}' if {row 'suite hash' for row in base} = {row 'suite hash' for row in cand}: raise SystemExit 'suite hash mismatch: baseline and candidate are not comparable' base cases = by case base, 'baseline' cand cases = by case cand, 'candidate' paired = sorted set base cases & set cand cases missing = sorted set base cases - set cand cases extra = sorted set cand cases - set base cases if missing: raise SystemExit f'candidate dropped cases: {missing :10 }' need = LIMITS 'min paired cases' if len paired < need: raise SystemExit f'need at least {need} paired cases, got {len paired }' ratios = {'wall': , 'compile': , 'tokens': } candidate queue = failures = for case id in paired: b = case stats base cases case id c = case stats cand cases case id candidate queue.append c 'queue' if not b 'passed' : failures.append f'{case id}: baseline did not pass' continue if not c 'passed' : failures.append f'{case id}: candidate regressed correctness' continue for key in ratios: ratios key .append ratio c key , b key if failures: report = {'verdict': 'FAIL', 'reason': 'correctness/coverage', 'failures': failures :20 } Path args.report .write text json.dumps report, indent=2 print json.dumps report, indent=2 return 1 checks = { 'p95 wall ratio': pct ratios 'wall' , 0.95 , 'p95 compile ratio': pct ratios 'compile' , 0.95 , 'median tokens ratio': median ratios 'tokens' , 'p90 case tokens ratio': pct ratios 'tokens' , 0.90 , 'queue wait p95 ms': pct candidate queue, 0.95 , } verdict = 'PASS' reasons = warnings = for key, limit in LIMITS.items : if key == 'min paired cases': continue if key == 'queue wait p95 ms': if checks key limit: warnings.append f'{key}={checks key :.0f} {limit}' elif checks key limit: verdict = 'FAIL' reasons.append f'{key}={checks key :.3f} {limit}' report = { 'verdict': verdict, 'checks': {key: round value, 3 for key, value in checks.items }, 'limits': LIMITS, 'paired cases': len paired , 'extra cases': extra, 'reasons': reasons, 'warnings': warnings, } Path args.report .write text json.dumps report, indent=2 print json.dumps report, indent=2 return 0 if verdict == 'PASS' else 1 if name == ' main ': sys.exit main A record is one JSON object per line. The minimum fields are shown as a Python template so the shape is explicit; the harness serializes this as JSONL. record = { 'run id': '2026-11-07T10:00:00Z', 'case id': 'cpp-vector-reserve-01', 'variant': 'baseline', 'suite hash': 'sha256:...', 'server class': 'free', 'passed': True, 'prompt tokens': 412, 'completion tokens': 188, 'compile ms': 310, 'exec ms': 42, 'queue wait ms': 1800, } The run command is intentionally boring: python3 budget gate.py --baseline baseline.jsonl --candidate candidate.jsonl --report budget-report.json A failing report should be readable in one glance: verdict=FAIL p95 wall ratio=1.31 limit 1.20 median tokens ratio=1.08 limit 1.15 queue wait p95 ms=41200 warn at 30000 | Check | Default gate | On breach | |---|---|---| | p95 wall ratio | 1.20 | fail | | p95 compile ratio | 1.25 | fail | | median tokens ratio | 1.15 | fail | | p90 case tokens ratio | 1.50 | fail | | queue wait p95 ms | 30000 | warn | | dropped case | any | fail | The ratios are paired by case, not calculated from aggregate totals. That matters. A candidate can keep the same total token count by trading a cheap case for an expensive one. The paired p90 catches tail hogs; the median catches broad quota burn; the wall and compile gates catch review latency that a pass-rate table never shows. The queue warning is separate for a reason. On a shared free server, a 40-second queue wait is a capacity signal, not a model-quality signal. It should not fail correctness, but it should stop a team from comparing a laptop baseline with a remote candidate and calling the difference a regression. server class . Record the prompt hash and model ID if the harness already has them; the budget gate only needs the fields above. queue wait ms ; shared capacity can add minutes that are not model execution time. MonkeyCode's open-source project is relevant to this workflow because the candidate side can run on its free model access and free server option. The operator-supplied offer at the time of writing includes a 10-million-token free allowance and a free server option. Quotas, capacity, and duration are not permanent; verify current terms before sizing a run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The cost model changes, but the gate does not. Free tokens still run out, and free shared servers still queue. The useful move is to keep baseline and candidate on the same server class, record queue wait separately, and treat a budget regression as a failed eval even when every test is green. The free allowance is enough to try the workflow on a small suite; check the current terms before scaling it. The gate has sharp edges. Token ratios are only comparable when both variants use the same tokenizer; otherwise compare billable units or quota consumption, not raw token counts. Compile time depends on flags, cache state, and machine class, so pin those or the ratio measures the host. Shared free capacity adds variance even with repeats; run baseline and candidate in the same window when possible. It also starts after correctness. If the candidate fails a golden case, the budget report is noise. If the suite has fewer than 20 paired cases, tail percentiles are unstable; use the gate as a directional check, not a hard merge block. Teams without per-case token and timing telemetry should add that telemetry before adopting the gate. A one-off patch review with no model choice does not need it. And no shared free server should be used to promise a hard real-time SLO; use the free option to find regressions, not to guarantee latency. Run the gate against the last two candidate models on a 20-case suite. A 1.2x p95 wall regression with a flat pass rate is a real finding. So is a 0.98 pass rate with a 2x token burn. Publish the budget report next to the eval score; the pair is more useful than another leaderboard screenshot.