{"slug": "benchmark-budgets-are-part-of-the-grader-a-paired-cost-and-latency-gate-for-ai-c", "title": "Benchmark Budgets Are Part of the Grader: A Paired Cost and Latency Gate for AI C++ Patches", "summary": "A developer published a paired cost-and-latency gate script, budget_gate.py, that sits after correctness tests in AI-generated C++ patch evaluations and flags when a candidate patch set becomes more expensive per accepted patch. The tool reads baseline and candidate JSONL files sharing a suite hash and case IDs, takes medians over repeats, requires every repeat to pass, and gates the p95 wall and compile ratios, median token ratio, and p90 per-case token ratio, with queue wait reported separately as a warning.", "body_md": "Consider a pull request that turns a 12-line C++ change into a 40-line patch. Every golden case passes. CI is green. Two days later the review queue tells a different story: the patch needed three retries per case, pushed p95 compile time up by a third, and burned a quarter more tokens than the model it replaced. The grader never noticed, because it only asked whether the tests passed.\n\nThat gap is the subject of this article. [The latest benchmarking challenge wrap-up](https://dev.to/devteam/congrats-to-the-kaggle-benchmarking-challenge-winners-2moh) is a useful reminder that leaderboards are snapshots, not budgets. For AI-generated C++ patches, a passing score has a second dimension: tokens, compile time, and wall-clock time per accepted patch.\n\nA correctness-only harness treats every passing patch as equal. A patch that compiles in 200 ms and one that compiles in 2 s both get a check mark. A patch that uses 800 tokens and one that uses 8,000 tokens both count as one success.\n\nThat is fine for a smoke test. It is not fine for a merge gate. Token allowances are finite, CI minutes are finite, and reviewer attention is finite. When a model swap improves pass rate by two points but doubles the tail latency, the eval has found a trade-off; the harness should report it instead of hiding it.\n\nThe gate below is deliberately small. It does not replace correctness tests, prompt lock files, or flaky-case triage. It sits after them and answers one narrow question: did the candidate patch set get more expensive per accepted patch?\n\nThe gate reads two JSONL files, one for the baseline and one for the candidate. Both must share a suite hash and the same case IDs. For each case, the script takes the median over repeats, requires every repeat to pass, and computes candidate-to-baseline ratios. It gates the 95th percentile of per-case wall and compile ratios, the median token ratio, and the 90th percentile of per-case token ratios. Queue wait is reported separately as a warning because shared capacity is noisy.\n\n``` bash\n#!/usr/bin/env python3\n'''budget_gate.py: paired cost and latency gate for AI C++ patch evals.'''\nimport argparse\nimport json\nimport sys\nfrom collections import defaultdict\nfrom pathlib import Path\nfrom statistics import median\n\nREQUIRED = {\n    'run_id', 'case_id', 'variant', 'suite_hash', 'server_class',\n    'passed', 'prompt_tokens', 'completion_tokens',\n    'compile_ms', 'exec_ms', 'queue_wait_ms',\n}\nLIMITS = {\n    'p95_wall_ratio': 1.20,\n    'p95_compile_ratio': 1.25,\n    'median_tokens_ratio': 1.15,\n    'p90_case_tokens_ratio': 1.50,\n    'queue_wait_p95_ms': 30_000,\n    'min_paired_cases': 20,\n}\n\ndef pct(values, q):\n    values = sorted(values)\n    if not values:\n        return 0.0\n    k = (len(values) - 1) * q\n    lo = int(k)\n    hi = min(lo + 1, len(values) - 1)\n    return values[lo] + (values[hi] - values[lo]) * (k - lo)\n\ndef load(path):\n    rows = []\n    for n, line in enumerate(Path(path).read_text().splitlines(), 1):\n        if not line.strip():\n            continue\n        try:\n            row = json.loads(line)\n        except json.JSONDecodeError as exc:\n            raise SystemExit(f'{path}:{n}: {exc}')\n        missing = REQUIRED - set(row)\n        if missing:\n            raise SystemExit(f'{path}:{n}: missing {sorted(missing)}')\n        rows.append(row)\n    return rows\n\ndef by_case(rows, variant):\n    grouped = defaultdict(list)\n    for row in rows:\n        if row['variant'] == variant:\n            grouped[row['case_id']].append(row)\n    return grouped\n\ndef case_stats(rows):\n    return {\n        'passed': all(bool(row['passed']) for row in rows),\n        'wall': median(float(row['compile_ms']) + float(row['exec_ms']) for row in rows),\n        'compile': median(float(row['compile_ms']) for row in rows),\n        'tokens': median(float(row['prompt_tokens']) + float(row['completion_tokens']) for row in rows),\n        'queue': median(float(row['queue_wait_ms']) for row in rows),\n    }\n\ndef ratio(candidate, baseline):\n    return candidate / baseline if baseline > 0 else float('inf')\n\ndef main():\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--baseline', required=True)\n    parser.add_argument('--candidate', required=True)\n    parser.add_argument('--report', default='budget-report.json')\n    args = parser.parse_args()\n    base = load(args.baseline)\n    cand = load(args.candidate)\n    for name, rows in (('baseline', base), ('candidate', cand)):\n        hashes = {row['suite_hash'] for row in rows}\n        if len(hashes) != 1:\n            raise SystemExit(f'{name}: mixed suite_hash {hashes}')\n    if {row['suite_hash'] for row in base} != {row['suite_hash'] for row in cand}:\n        raise SystemExit('suite_hash mismatch: baseline and candidate are not comparable')\n    base_cases = by_case(base, 'baseline')\n    cand_cases = by_case(cand, 'candidate')\n    paired = sorted(set(base_cases) & set(cand_cases))\n    missing = sorted(set(base_cases) - set(cand_cases))\n    extra = sorted(set(cand_cases) - set(base_cases))\n    if missing:\n        raise SystemExit(f'candidate dropped cases: {missing[:10]}')\n    need = LIMITS['min_paired_cases']\n    if len(paired) < need:\n        raise SystemExit(f'need at least {need} paired cases, got {len(paired)}')\n    ratios = {'wall': [], 'compile': [], 'tokens': []}\n    candidate_queue = []\n    failures = []\n    for case_id in paired:\n        b = case_stats(base_cases[case_id])\n        c = case_stats(cand_cases[case_id])\n        candidate_queue.append(c['queue'])\n        if not b['passed']:\n            failures.append(f'{case_id}: baseline did not pass')\n            continue\n        if not c['passed']:\n            failures.append(f'{case_id}: candidate regressed correctness')\n            continue\n        for key in ratios:\n            ratios[key].append(ratio(c[key], b[key]))\n    if failures:\n        report = {'verdict': 'FAIL', 'reason': 'correctness/coverage', 'failures': failures[:20]}\n        Path(args.report).write_text(json.dumps(report, indent=2))\n        print(json.dumps(report, indent=2))\n        return 1\n    checks = {\n        'p95_wall_ratio': pct(ratios['wall'], 0.95),\n        'p95_compile_ratio': pct(ratios['compile'], 0.95),\n        'median_tokens_ratio': median(ratios['tokens']),\n        'p90_case_tokens_ratio': pct(ratios['tokens'], 0.90),\n        'queue_wait_p95_ms': pct(candidate_queue, 0.95),\n    }\n    verdict = 'PASS'\n    reasons = []\n    warnings = []\n    for key, limit in LIMITS.items():\n        if key == 'min_paired_cases':\n            continue\n        if key == 'queue_wait_p95_ms':\n            if checks[key] > limit:\n                warnings.append(f'{key}={checks[key]:.0f} > {limit}')\n        elif checks[key] > limit:\n            verdict = 'FAIL'\n            reasons.append(f'{key}={checks[key]:.3f} > {limit}')\n    report = {\n        'verdict': verdict,\n        'checks': {key: round(value, 3) for key, value in checks.items()},\n        'limits': LIMITS,\n        'paired_cases': len(paired),\n        'extra_cases': extra,\n        'reasons': reasons,\n        'warnings': warnings,\n    }\n    Path(args.report).write_text(json.dumps(report, indent=2))\n    print(json.dumps(report, indent=2))\n    return 0 if verdict == 'PASS' else 1\n\nif __name__ == '__main__':\n    sys.exit(main())\n```\n\nA record is one JSON object per line. The minimum fields are shown as a Python template so the shape is explicit; the harness serializes this as JSONL.\n\n```\nrecord = {\n    'run_id': '2026-11-07T10:00:00Z',\n    'case_id': 'cpp-vector-reserve-01',\n    'variant': 'baseline',\n    'suite_hash': 'sha256:...',\n    'server_class': 'free',\n    'passed': True,\n    'prompt_tokens': 412,\n    'completion_tokens': 188,\n    'compile_ms': 310,\n    'exec_ms': 42,\n    'queue_wait_ms': 1800,\n}\n```\n\nThe run command is intentionally boring:\n\n```\npython3 budget_gate.py --baseline baseline.jsonl --candidate candidate.jsonl --report budget-report.json\n```\n\nA failing report should be readable in one glance:\n\n```\nverdict=FAIL\np95_wall_ratio=1.31 (limit 1.20)\nmedian_tokens_ratio=1.08 (limit 1.15)\nqueue_wait_p95_ms=41200 (warn at 30000)\n```\n\n| Check | Default gate | On breach | \n|---|---|---|\n| `p95_wall_ratio` | `> 1.20` | fail | \n| `p95_compile_ratio` | `> 1.25` | fail | \n| `median_tokens_ratio` | `> 1.15` | fail | \n| `p90_case_tokens_ratio` | `> 1.50` | fail | \n| `queue_wait_p95_ms` | `> 30000` | warn | \n| dropped case | any | fail | \n\nThe ratios are paired by case, not calculated from aggregate totals. That matters. A candidate can keep the same total token count by trading a cheap case for an expensive one. The paired p90 catches tail hogs; the median catches broad quota burn; the wall and compile gates catch review latency that a pass-rate table never shows.\n\nThe queue warning is separate for a reason. On a shared free server, a 40-second queue wait is a capacity signal, not a model-quality signal. It should not fail correctness, but it should stop a team from comparing a laptop baseline with a remote candidate and calling the difference a regression.\n\n`server_class`. Record the prompt hash and model ID if the harness already has them; the budget gate only needs the fields above.`queue_wait_ms`; shared capacity can add minutes that are not model execution time.\nMonkeyCode's open-source project is relevant to this workflow because the candidate side can run on its free model access and free server option. The operator-supplied offer at the time of writing includes a 10-million-token free allowance and a free server option. Quotas, capacity, and duration are not permanent; verify current terms before sizing a run.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nThe cost model changes, but the gate does not. Free tokens still run out, and free shared servers still queue. The useful move is to keep baseline and candidate on the same server class, record queue wait separately, and treat a budget regression as a failed eval even when every test is green. The free allowance is enough to try the workflow on a small suite; check the current terms before scaling it.\n\nThe gate has sharp edges. Token ratios are only comparable when both variants use the same tokenizer; otherwise compare billable units or quota consumption, not raw token counts. Compile time depends on flags, cache state, and machine class, so pin those or the ratio measures the host. Shared free capacity adds variance even with repeats; run baseline and candidate in the same window when possible.\n\nIt also starts after correctness. If the candidate fails a golden case, the budget report is noise. If the suite has fewer than 20 paired cases, tail percentiles are unstable; use the gate as a directional check, not a hard merge block. Teams without per-case token and timing telemetry should add that telemetry before adopting the gate. A one-off patch review with no model choice does not need it. And no shared free server should be used to promise a hard real-time SLO; use the free option to find regressions, not to guarantee latency.\n\nRun the gate against the last two candidate models on a 20-case suite. A 1.2x p95 wall regression with a flat pass rate is a real finding. So is a 0.98 pass rate with a 2x token burn. Publish the budget report next to the eval score; the pair is more useful than another leaderboard screenshot.", "url": "https://wpnews.pro/news/benchmark-budgets-are-part-of-the-grader-a-paired-cost-and-latency-gate-for-ai-c", "canonical_source": "https://dev.to/devcpp_268/benchmark-budgets-are-part-of-the-grader-a-paired-cost-and-latency-gate-for-ai-c-patches-404k", "published_at": "2026-11-06 16:43:18+00:00", "updated_at": "2026-11-06 16:53:47.876146+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "ai-agents"], "entities": ["budget_gate.py", "Kaggle", "dev.to"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/benchmark-budgets-are-part-of-the-grader-a-paired-cost-and-latency-gate-for-ai-c", "markdown": "https://wpnews.pro/news/benchmark-budgets-are-part-of-the-grader-a-paired-cost-and-latency-gate-for-ai-c.md", "text": "https://wpnews.pro/news/benchmark-budgets-are-part-of-the-grader-a-paired-cost-and-latency-gate-for-ai-c.txt", "jsonld": "https://wpnews.pro/news/benchmark-budgets-are-part-of-the-grader-a-paired-cost-and-latency-gate-for-ai-c.jsonld"}}