Benchmark Budgets Are Part of the Grader: A Paired Cost and Latency Gate for AI C++ Patches

Consider a pull request that turns a 12-line C++ change into a 40-line patch. Every golden case passes. CI is green. Two days later the review queue tells a different story: the patch needed three retries per case, pushed p95 compile time up by a third, and burned a quarter more tokens than the model it replaced. The grader never noticed, because it only asked whether the tests passed.

That gap is the subject of this article. The latest benchmarking challenge wrap-up is a useful reminder that leaderboards are snapshots, not budgets. For AI-generated C++ patches, a passing score has a second dimension: tokens, compile time, and wall-clock time per accepted patch.

A correctness-only harness treats every passing patch as equal. A patch that compiles in 200 ms and one that compiles in 2 s both get a check mark. A patch that uses 800 tokens and one that uses 8,000 tokens both count as one success.

That is fine for a smoke test. It is not fine for a merge gate. Token allowances are finite, CI minutes are finite, and reviewer attention is finite. When a model swap improves pass rate by two points but doubles the tail latency, the eval has found a trade-off; the harness should report it instead of hiding it.

The gate below is deliberately small. It does not replace correctness tests, prompt lock files, or flaky-case triage. It sits after them and answers one narrow question: did the candidate patch set get more expensive per accepted patch?

The gate reads two JSONL files, one for the baseline and one for the candidate. Both must share a suite hash and the same case IDs. For each case, the script takes the median over repeats, requires every repeat to pass, and computes candidate-to-baseline ratios. It gates the 95th percentile of per-case wall and compile ratios, the median token ratio, and the 90th percentile of per-case token ratios. Queue wait is reported separately as a warning because shared capacity is noisy.

#!/usr/bin/env python3
'''budget_gate.py: paired cost and latency gate for AI C++ patch evals.'''
import argparse
import json
import sys
from collections import defaultdict
from pathlib import Path
from statistics import median

REQUIRED = {
    'run_id', 'case_id', 'variant', 'suite_hash', 'server_class',
    'passed', 'prompt_tokens', 'completion_tokens',
    'compile_ms', 'exec_ms', 'queue_wait_ms',
}
LIMITS = {
    'p95_wall_ratio': 1.20,
    'p95_compile_ratio': 1.25,
    'median_tokens_ratio': 1.15,
    'p90_case_tokens_ratio': 1.50,
    'queue_wait_p95_ms': 30_000,
    'min_paired_cases': 20,
}

def pct(values, q):
    values = sorted(values)
    if not values:
        return 0.0
    k = (len(values) - 1) * q
    lo = int(k)
    hi = min(lo + 1, len(values) - 1)
    return values[lo] + (values[hi] - values[lo]) * (k - lo)

def load(path):
    rows = []
    for n, line in enumerate(Path(path).read_text().splitlines(), 1):
        if not line.strip():
            continue
        try:
            row = json.loads(line)
        except json.JSONDecodeError as exc:
            raise SystemExit(f'{path}:{n}: {exc}')
        missing = REQUIRED - set(row)
        if missing:
            raise SystemExit(f'{path}:{n}: missing {sorted(missing)}')
        rows.append(row)
    return rows

def by_case(rows, variant):
    grouped = defaultdict(list)
    for row in rows:
        if row['variant'] == variant:
            grouped[row['case_id']].append(row)
    return grouped

def case_stats(rows):
    return {
        'passed': all(bool(row['passed']) for row in rows),
        'wall': median(float(row['compile_ms']) + float(row['exec_ms']) for row in rows),
        'compile': median(float(row['compile_ms']) for row in rows),
        'tokens': median(float(row['prompt_tokens']) + float(row['completion_tokens']) for row in rows),
        'queue': median(float(row['queue_wait_ms']) for row in rows),
    }

def ratio(candidate, baseline):
    return candidate / baseline if baseline > 0 else float('inf')

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--baseline', required=True)
    parser.add_argument('--candidate', required=True)
    parser.add_argument('--report', default='budget-report.json')
    args = parser.parse_args()
    base = load(args.baseline)
    cand = load(args.candidate)
    for name, rows in (('baseline', base), ('candidate', cand)):
        hashes = {row['suite_hash'] for row in rows}
        if len(hashes) != 1:
            raise SystemExit(f'{name}: mixed suite_hash {hashes}')
    if {row['suite_hash'] for row in base} != {row['suite_hash'] for row in cand}:
        raise SystemExit('suite_hash mismatch: baseline and candidate are not comparable')
    base_cases = by_case(base, 'baseline')
    cand_cases = by_case(cand, 'candidate')
    paired = sorted(set(base_cases) & set(cand_cases))
    missing = sorted(set(base_cases) - set(cand_cases))
    extra = sorted(set(cand_cases) - set(base_cases))
    if missing:
        raise SystemExit(f'candidate dropped cases: {missing[:10]}')
    need = LIMITS['min_paired_cases']
    if len(paired) < need:
        raise SystemExit(f'need at least {need} paired cases, got {len(paired)}')
    ratios = {'wall': [], 'compile': [], 'tokens': []}
    candidate_queue = []
    failures = []
    for case_id in paired:
        b = case_stats(base_cases[case_id])
        c = case_stats(cand_cases[case_id])
        candidate_queue.append(c['queue'])
        if not b['passed']:
            failures.append(f'{case_id}: baseline did not pass')
            continue
        if not c['passed']:
            failures.append(f'{case_id}: candidate regressed correctness')
            continue
        for key in ratios:
            ratios[key].append(ratio(c[key], b[key]))
    if failures:
        report = {'verdict': 'FAIL', 'reason': 'correctness/coverage', 'failures': failures[:20]}
        Path(args.report).write_text(json.dumps(report, indent=2))
        print(json.dumps(report, indent=2))
        return 1
    checks = {
        'p95_wall_ratio': pct(ratios['wall'], 0.95),
        'p95_compile_ratio': pct(ratios['compile'], 0.95),
        'median_tokens_ratio': median(ratios['tokens']),
        'p90_case_tokens_ratio': pct(ratios['tokens'], 0.90),
        'queue_wait_p95_ms': pct(candidate_queue, 0.95),
    }
    verdict = 'PASS'
    reasons = []
    warnings = []
    for key, limit in LIMITS.items():
        if key == 'min_paired_cases':
            continue
        if key == 'queue_wait_p95_ms':
            if checks[key] > limit:
                warnings.append(f'{key}={checks[key]:.0f} > {limit}')
        elif checks[key] > limit:
            verdict = 'FAIL'
            reasons.append(f'{key}={checks[key]:.3f} > {limit}')
    report = {
        'verdict': verdict,
        'checks': {key: round(value, 3) for key, value in checks.items()},
        'limits': LIMITS,
        'paired_cases': len(paired),
        'extra_cases': extra,
        'reasons': reasons,
        'warnings': warnings,
    }
    Path(args.report).write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))
    return 0 if verdict == 'PASS' else 1

if __name__ == '__main__':
    sys.exit(main())

A record is one JSON object per line. The minimum fields are shown as a Python template so the shape is explicit; the harness serializes this as JSONL.

record = {
    'run_id': '2026-11-07T10:00:00Z',
    'case_id': 'cpp-vector-reserve-01',
    'variant': 'baseline',
    'suite_hash': 'sha256:...',
    'server_class': 'free',
    'passed': True,
    'prompt_tokens': 412,
    'completion_tokens': 188,
    'compile_ms': 310,
    'exec_ms': 42,
    'queue_wait_ms': 1800,
}

The run command is intentionally boring:

python3 budget_gate.py --baseline baseline.jsonl --candidate candidate.jsonl --report budget-report.json

A failing report should be readable in one glance:

verdict=FAIL
p95_wall_ratio=1.31 (limit 1.20)
median_tokens_ratio=1.08 (limit 1.15)
queue_wait_p95_ms=41200 (warn at 30000)

The ratios are paired by case, not calculated from aggregate totals. That matters. A candidate can keep the same total token count by trading a cheap case for an expensive one. The paired p90 catches tail hogs; the median catches broad quota burn; the wall and compile gates catch review latency that a pass-rate table never shows.

The queue warning is separate for a reason. On a shared free server, a 40-second queue wait is a capacity signal, not a model-quality signal. It should not fail correctness, but it should stop a team from comparing a laptop baseline with a remote candidate and calling the difference a regression.

server_class. Record the prompt hash and model ID if the harness already has them; the budget gate only needs the fields above.queue_wait_ms; shared capacity can add minutes that are not model execution time. MonkeyCode's open-source project is relevant to this workflow because the candidate side can run on its free model access and free server option. The operator-supplied offer at the time of writing includes a 10-million-token free allowance and a free server option. Quotas, capacity, and duration are not permanent; verify current terms before sizing a run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The cost model changes, but the gate does not. Free tokens still run out, and free shared servers still queue. The useful move is to keep baseline and candidate on the same server class, record queue wait separately, and treat a budget regression as a failed eval even when every test is green. The free allowance is enough to try the workflow on a small suite; check the current terms before scaling it.

The gate has sharp edges. Token ratios are only comparable when both variants use the same tokenizer; otherwise compare billable units or quota consumption, not raw token counts. Compile time depends on flags, cache state, and machine class, so pin those or the ratio measures the host. Shared free capacity adds variance even with repeats; run baseline and candidate in the same window when possible.

It also starts after correctness. If the candidate fails a golden case, the budget report is noise. If the suite has fewer than 20 paired cases, tail percentiles are unstable; use the gate as a directional check, not a hard merge block. Teams without per-case token and timing telemetry should add that telemetry before adopting the gate. A one-off patch review with no model choice does not need it. And no shared free server should be used to promise a hard real-time SLO; use the free option to find regressions, not to guarantee latency.

Run the gate against the last two candidate models on a 20-case suite. A 1.2x p95 wall regression with a flat pass rate is a real finding. So is a 0.98 pass rate with a 2x token burn. Publish the budget report next to the eval score; the pair is more useful than another leaderboard screenshot.