If 30% of Coding Tasks May Be Broken, Your Leaderboard Needs an Uncertainty Budget

OpenAI published an audit of SWE-Bench Pro on July 8, 2026, estimating that roughly 30% of its coding tasks are broken. The company proposed a methodology for reporting benchmark scores with an uncertainty budget, including sensitivity bounds for unresolved task validity. The approach requires versioning task validity, preserving disputed cases, and publishing how conclusions change across plausible denominators.

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial. Primary source: OpenAI, “Separating signal from noise in coding evaluations” https://openai.com/index/separating-signal-from-noise-coding-evaluations/ . The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators. task validity: unreviewed | valid | broken | disputed model result: pass | fail | infrastructure error | missing Never convert infrastructure error to model failure without reporting that policy. Never delete broken tasks while retaining an old score label. A row needs provenance: { "task id": "repo-issue-17", "dataset revision": "sha256:...", "harness revision": "git:...", "model config": "immutable-config-id", "validity": "disputed", "result": "pass", "review revision": 3, "evidence": "fixture.log", "review.json" } Let: P v , N v : passes and total among reviewed-valid tasks; P a , N a : passes and total across all attempted tasks; D : disputed tasks.Report: valid-only score = P v / N v all-attempted score = P a / N a uncertainty interval = score if every disputed task hurts conclusion .. score if every disputed task helps conclusion This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity. python /usr/bin/env python3 import json, sys rows = json.loads line for line in open sys.argv 1 if line.strip valid = r for r in rows if r "validity" == "valid" disputed = r for r in rows if r "validity" in "unreviewed", "disputed" attempted = r for r in rows if r "result" in "pass", "fail" rate = lambda passed, total: passed / total if total else float "nan" valid pass = sum r "result" == "pass" for r in valid all pass = sum r "result" == "pass" for r in attempted Bounds ask how unresolved validity could change the valid-only denominator. low = rate valid pass, len valid + len disputed high = rate valid pass + len disputed , len valid + len disputed print json.dumps { "valid": {"pass": valid pass, "n": len valid , "rate": rate valid pass, len valid }, "attempted": {"pass": all pass, "n": len attempted , "rate": rate all pass, len attempted }, "disputed n": len disputed , "validity sensitivity": low, high }, indent=2 Run it against an immutable JSONL export: python3 sensitivity.py results.jsonl The calculator deliberately makes a pessimistic simplification: every unresolved task can move the numerator one way or the other. A real audit can narrow the interval by classifying tasks and preserving reviewer disagreement. Before trusting the pipeline, simulate: broken to valid after score publication.Invariants: - Raw attempts are append-only. - Validity revisions never overwrite earlier reviews. - Every published score names dataset, harness, and review revisions. - Reclassification triggers deterministic recomputation. - Infrastructure errors have their own count. - Pairwise model comparisons use the same eligible task set. That last invariant matters. Comparing Model A on 600 valid tasks with Model B on 650 partly different tasks introduces a denominator change disguised as capability change. Suppose two models differ by two percentage points, but either model's score moves by five points under plausible task-validity decisions. The benchmark cannot support the ranking on its own. Use a rule such as: accept model change only if: common valid tasks: " = 500" infrastructure error rate: "< 1%" disputed fraction: "< 5%" observed margin: " validity sensitivity + run variance" production canary: "passes workload-specific gate" Thresholds are policy choices, not universal constants. Name the owner and revisit date. A cleaner coding score does not prove lower incident rates, faster review, or better performance in your repository. Pair public benchmark evidence with a private, permission-safe canary containing your languages, tests, dependency graph, and review rules. The audit's most useful lesson is architectural: a benchmark task is not merely an array element. It is a versioned claim that the input, oracle, environment, and scoring path can distinguish capability from noise. If your preferred model changes when disputed tasks are removed, do you publish the new winner—or publish that the ranking is unresolved?