If 30% of Coding Tasks May Be Broken, Your Leaderboard Needs an Uncertainty Budget

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial.

Primary source: OpenAI, “Separating signal from noise in coding evaluations”.

The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators.

task validity: unreviewed | valid | broken | disputed
model result:  pass | fail | infrastructure_error | missing

Never convert infrastructure_error

to model failure without reporting that policy. Never delete broken tasks while retaining an old score label.

A row needs provenance:

{
  "task_id": "repo-issue-17",
  "dataset_revision": "sha256:...",
  "harness_revision": "git:...",
  "model_config": "immutable-config-id",
  "validity": "disputed",
  "result": "pass",
  "review_revision": 3,
  "evidence": ["fixture.log", "review.json"]
}

Let:

P_v

, N_v

: passes and total among reviewed-valid tasks;P_a

, N_a

: passes and total across all attempted tasks;D

: disputed tasks.Report:

valid-only score = P_v / N_v
all-attempted score = P_a / N_a
uncertainty interval = score if every disputed task hurts conclusion
                       .. score if every disputed task helps conclusion

This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity.

#!/usr/bin/env python3
import json, sys

rows = [json.loads(line) for line in open(sys.argv[1]) if line.strip()]
valid = [r for r in rows if r["validity"] == "valid"]
disputed = [r for r in rows if r["validity"] in ("unreviewed", "disputed")]
attempted = [r for r in rows if r["result"] in ("pass", "fail")]

rate = lambda passed, total: passed / total if total else float("nan")
valid_pass = sum(r["result"] == "pass" for r in valid)
all_pass = sum(r["result"] == "pass" for r in attempted)

low = rate(valid_pass, len(valid) + len(disputed))
high = rate(valid_pass + len(disputed), len(valid) + len(disputed))

print(json.dumps({
  "valid": {"pass": valid_pass, "n": len(valid), "rate": rate(valid_pass, len(valid))},
  "attempted": {"pass": all_pass, "n": len(attempted), "rate": rate(all_pass, len(attempted))},
  "disputed_n": len(disputed),
  "validity_sensitivity": [low, high]
}, indent=2))

Run it against an immutable JSONL export:

python3 sensitivity.py results.jsonl

The calculator deliberately makes a pessimistic simplification: every unresolved task can move the numerator one way or the other. A real audit can narrow the interval by classifying tasks and preserving reviewer disagreement.

Before trusting the pipeline, simulate:

broken

to valid

after score publication.Invariants:

- Raw attempts are append-only.
- Validity revisions never overwrite earlier reviews.
- Every published score names dataset, harness, and review revisions.
- Reclassification triggers deterministic recomputation.
- Infrastructure errors have their own count.
- Pairwise model comparisons use the same eligible task set.

That last invariant matters. Comparing Model A on 600 valid tasks with Model B on 650 partly different tasks introduces a denominator change disguised as capability change.

Suppose two models differ by two percentage points, but either model's score moves by five points under plausible task-validity decisions. The benchmark cannot support the ranking on its own.

Use a rule such as:

accept_model_change_only_if:
  common_valid_tasks: ">= 500"
  infrastructure_error_rate: "< 1%"
  disputed_fraction: "< 5%"
  observed_margin: "> validity_sensitivity + run_variance"
  production_canary: "passes workload-specific gate"

Thresholds are policy choices, not universal constants. Name the owner and revisit date.

A cleaner coding score does not prove lower incident rates, faster review, or better performance in your repository. Pair public benchmark evidence with a private, permission-safe canary containing your languages, tests, dependency graph, and review rules.

The audit's most useful lesson is architectural: a benchmark task is not merely an array element. It is a versioned claim that the input, oracle, environment, and scoring path can distinguish capability from noise.

If your preferred model changes when disputed tasks are removed, do you publish the new winner—or publish that the ranking is unresolved?