{"slug": "if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget", "title": "If 30% of Coding Tasks May Be Broken, Your Leaderboard Needs an Uncertainty Budget", "summary": "OpenAI published an audit of SWE-Bench Pro on July 8, 2026, estimating that roughly 30% of its coding tasks are broken. The company proposed a methodology for reporting benchmark scores with an uncertainty budget, including sensitivity bounds for unresolved task validity. The approach requires versioning task validity, preserving disputed cases, and publishing how conclusions change across plausible denominators.", "body_md": "OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial.\n\nPrimary source: [OpenAI, “Separating signal from noise in coding evaluations”](https://openai.com/index/separating-signal-from-noise-coding-evaluations/).\n\nThe operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators.\n\n```\ntask validity: unreviewed | valid | broken | disputed\nmodel result:  pass | fail | infrastructure_error | missing\n```\n\nNever convert `infrastructure_error`\n\nto model failure without reporting that policy. Never delete broken tasks while retaining an old score label.\n\nA row needs provenance:\n\n```\n{\n  \"task_id\": \"repo-issue-17\",\n  \"dataset_revision\": \"sha256:...\",\n  \"harness_revision\": \"git:...\",\n  \"model_config\": \"immutable-config-id\",\n  \"validity\": \"disputed\",\n  \"result\": \"pass\",\n  \"review_revision\": 3,\n  \"evidence\": [\"fixture.log\", \"review.json\"]\n}\n```\n\nLet:\n\n`P_v`\n\n, `N_v`\n\n: passes and total among reviewed-valid tasks;`P_a`\n\n, `N_a`\n\n: passes and total across all attempted tasks;`D`\n\n: disputed tasks.Report:\n\n```\nvalid-only score = P_v / N_v\nall-attempted score = P_a / N_a\nuncertainty interval = score if every disputed task hurts conclusion\n                       .. score if every disputed task helps conclusion\n```\n\nThis interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity.\n\n``` python\n#!/usr/bin/env python3\nimport json, sys\n\nrows = [json.loads(line) for line in open(sys.argv[1]) if line.strip()]\nvalid = [r for r in rows if r[\"validity\"] == \"valid\"]\ndisputed = [r for r in rows if r[\"validity\"] in (\"unreviewed\", \"disputed\")]\nattempted = [r for r in rows if r[\"result\"] in (\"pass\", \"fail\")]\n\nrate = lambda passed, total: passed / total if total else float(\"nan\")\nvalid_pass = sum(r[\"result\"] == \"pass\" for r in valid)\nall_pass = sum(r[\"result\"] == \"pass\" for r in attempted)\n\n# Bounds ask how unresolved validity could change the valid-only denominator.\nlow = rate(valid_pass, len(valid) + len(disputed))\nhigh = rate(valid_pass + len(disputed), len(valid) + len(disputed))\n\nprint(json.dumps({\n  \"valid\": {\"pass\": valid_pass, \"n\": len(valid), \"rate\": rate(valid_pass, len(valid))},\n  \"attempted\": {\"pass\": all_pass, \"n\": len(attempted), \"rate\": rate(all_pass, len(attempted))},\n  \"disputed_n\": len(disputed),\n  \"validity_sensitivity\": [low, high]\n}, indent=2))\n```\n\nRun it against an immutable JSONL export:\n\n```\npython3 sensitivity.py results.jsonl\n```\n\nThe calculator deliberately makes a pessimistic simplification: every unresolved task can move the numerator one way or the other. A real audit can narrow the interval by classifying tasks and preserving reviewer disagreement.\n\nBefore trusting the pipeline, simulate:\n\n`broken`\n\nto `valid`\n\nafter score publication.Invariants:\n\n```\n- Raw attempts are append-only.\n- Validity revisions never overwrite earlier reviews.\n- Every published score names dataset, harness, and review revisions.\n- Reclassification triggers deterministic recomputation.\n- Infrastructure errors have their own count.\n- Pairwise model comparisons use the same eligible task set.\n```\n\nThat last invariant matters. Comparing Model A on 600 valid tasks with Model B on 650 partly different tasks introduces a denominator change disguised as capability change.\n\nSuppose two models differ by two percentage points, but either model's score moves by five points under plausible task-validity decisions. The benchmark cannot support the ranking on its own.\n\nUse a rule such as:\n\n```\naccept_model_change_only_if:\n  common_valid_tasks: \">= 500\"\n  infrastructure_error_rate: \"< 1%\"\n  disputed_fraction: \"< 5%\"\n  observed_margin: \"> validity_sensitivity + run_variance\"\n  production_canary: \"passes workload-specific gate\"\n```\n\nThresholds are policy choices, not universal constants. Name the owner and revisit date.\n\nA cleaner coding score does not prove lower incident rates, faster review, or better performance in your repository. Pair public benchmark evidence with a private, permission-safe canary containing your languages, tests, dependency graph, and review rules.\n\nThe audit's most useful lesson is architectural: a benchmark task is not merely an array element. It is a versioned claim that the input, oracle, environment, and scoring path can distinguish capability from noise.\n\nIf your preferred model changes when disputed tasks are removed, do you publish the new winner—or publish that the ranking is unresolved?", "url": "https://wpnews.pro/news/if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget", "canonical_source": "https://dev.to/robinzzz/if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget-4992", "published_at": "2026-07-17 06:56:18+00:00", "updated_at": "2026-07-17 07:01:05.093109+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety", "developer-tools"], "entities": ["OpenAI", "SWE-Bench Pro"], "alternates": {"html": "https://wpnews.pro/news/if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget", "markdown": "https://wpnews.pro/news/if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget.md", "text": "https://wpnews.pro/news/if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget.txt", "jsonld": "https://wpnews.pro/news/if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget.jsonld"}}