Benchmarks are more broken than we could have imagined

An audit by Horizon of 20 public task datasets in the Harbor hub found 29 confirmed broken tasks out of 5,241 scanned, with failures that often made models look better rather than worse, according to the company's first vendor quality index release. The audit scanned all 5,241 tasks, checked 239 more closely, and reviewed 34 one by one; contamination, incomplete tests, and gameable grading accounted for 39 of the 54 category placements. The findings follow Epoch AI's recent audit of 15 benchmarks, nine of which it labeled flawed.

Epoch AI recently audited 15 benchmarks and labeled nine of them flawed https://x.com/YafahEdelman/status/2100718900262707396 . Its reviews found leaked answers, reward hacks, incomplete tests, and verifier failures. That was alarming enough. We looked one layer lower: at public task datasets from major vendors in the Harbor hub. For each task, we asked whether the instruction, environment, reference solution, and grader actually agreed. Across 20 datasets and 5,241 tasks, we confirmed 29 broken tasks. Many of the failures did not make models look worse. They made models look better by exposing answers, testing only part of the requested work, or rewarding shortcuts. Those numbers come from different stages of the audit. We scanned all 5,241 tasks, checked 239 more closely, and reviewed 34 one by one. The 29 confirmed broken tasks come from the audit as a whole—not only the 34-task manual review. This is the first release of a continuing audit and the starting point for Horizon’s vendor quality index. Five ways a benchmark breaks The failures were different, but not random. Every confirmed finding fit into one of five categories. - Answer contamination: The task environment contains the answer. For example, an agent can recover the finished fix from Git history instead of solving the bug. - Incomplete tests: The grader checks less than the instruction requires. For example, a task asks for changes across four parts of a streaming system while the tests exercise only one package. - Trivially gameable grading: A model can score without doing the work. For example, changing an editable dictionary makes the same fabricated answer go from zero to full marks. - Spec-to-test or capability mismatch: The instruction and grader disagree, or passing does not demonstrate the skill the task claims to test. For example, two parts of a grading configuration require different financial values for the same answer. - Oracle failure or flaky grading: The published solution fails, or the score depends on something unstable. For example, the author’s own solution receives zero after an external database changes. A task can fail in more than one way, so these counts overlap. Most of the failures made models look better, not worse Contamination, incomplete tests, and gameable grading account for 39 of the 54 category placements in this audit. These failures tend to give marks away. A model can appear more capable because it found an answer in the environment, completed the part that happened to be tested, or discovered a shortcut the grader accepted. The result is not simply noise. It is usually flattering noise. The mistakes were not evenly distributed The same problem did not appear everywhere. Contamination dominated the Scale AI sample we reviewed. The Terminal Bench sample had more incomplete tests, shortcuts, and grading failures. The OpenThoughts findings included editable grader inputs and reference solutions that did not solve their own tasks. That does not mean every dataset from those vendors has the same problem. It means the audited samples failed in different ways. A single overall flaw rate would hide the reason. These are counts of confirmed findings in the samples we audited, not vendor-wide failure rates. A blank cell means we did not confirm that failure type here. It does not prove that none exists. What a broken task looks like Five examples show how a task can return a clean score for the wrong reason. The answer was already in the image Scale AI / SWE-bench Pro — instance ansible ansible-0ea40e09 The task asked the model to repair an Ansible bug. The task image still contained the finished fix in its Git history. We recovered the fix with the network disabled, applied it, and ran the task’s real verifier. It returned full marks: 16 out of 16 tests. The model did not need to solve the bug. It only needed to find the answer that had shipped with it. One file got full marks Scale AI / SWE-bench Pro — navidrome navidrome-812dc209 The task asked for a time offset to pass through an entire streaming path: the cache key, two endpoints, and the templates. We changed one file. Two other relevant packages did not compile. The submission still received full marks because the grader exercised only the FFmpeg package. The task described a system-wide change. The score measured a command string. Change the dictionary, change the score OpenThoughts — word-derangement-mapping The grader read its dictionary from the same workspace the solver could edit. We submitted the same fabricated answer twice. It scored zero with the original dictionary and full marks after the dictionary was replaced. Nothing about the answer improved. We changed what the grader trusted. The grader expected two different answers Mercor / Apex Agents — world224-sk-task03-dfcbb713 One part of the grading configuration expected sponsor equity of $28,137 and an IRR of 20.4%. Another accepted only $28,517 and 20.8%. A response could follow one published expectation and fail the other. This is not a hard task. It is a task with two definitions of correct. The published solution scored zero Terminal Bench — protein-assembly We ran the author’s own solution five times, including three runs against the version served by the hub during this audit. It scored zero every time. The external protein data had changed. The tests still expected the older representation, leaving no valid sequence that could satisfy both the data and the grader. A model cannot solve a task whose accepted answer no longer exists. A shortcut does not need to be used to be a defect It is possible that a model completes one of the contaminated tasks without looking through Git history. Some probably do. That does not make the package safe to use. Once the answer is reachable, the same score can mean two different things. One model may understand the code and repair it. Another may inspect the environment, find the hidden commit, and copy the patch. The grader cannot tell them apart. This matters more as agents improve. Capable models explore their environments, inspect repository history, read build artifacts, and look for the shortest reliable path to a result. Finding a shortcut is often useful behavior in real work. It is a problem when the benchmark claims the resulting score measures a different skill. A benchmark should survive the models it is intended to test. We should not have to hope that the model misses the answer. A ranking needs more than a flaw count It would be easy to sort vendors by the number of problems we found and stop there. That would hide too much. The index will show five things together: - Initial flag rate: The share of tasks flagged as flawed in a comparable random sample. - Severity: Whether the defects are narrow, materially distort a score, or make the task impossible to interpret. - Failure breadth: How many different failure categories appeared. Several unrelated kinds of failure suggest a broader quality-control problem. - Evidence strength: Whether the finding was reproduced with the real verifier, tested directly against the grader, or confirmed from the published files. - Review confidence: The sample size, random selection, inspectability, and checks we were able to run. One task with three defects still counts once in the rate. Its most serious defect sets its severity, while the separate breadth measure records how many kinds of problems appeared. This avoids inflating the score while still distinguishing an isolated mismatch from a dataset with leakage, shortcuts, incomplete tests, and broken solutions. Some vendors have one dataset in the audit. Some have several. Some graders could be run directly. Others depend on paid judges we did not have access to. For those vendors, we report the initial screening separately from confirmed findings. An unrun grader is not a clean result, but it is also not enough for a precise vendor-wide failure rate. The useful comparison is not simply who has the fewest visible mistakes. It is who consistently publishes tasks whose instructions, environments, solutions, and graders agree—and how certain we are about that conclusion. The current provisional risk score weights the share of sampled tasks flagged as flawed at 30%, the worst confirmed severity at 40%, and confirmed failure breadth at 30%. Evidence strength and review confidence remain separate. A vendor without a comparable random sample receives a lower-bound score based on confirmed evidence, marked with a plus sign. View the preliminary data quality index https://www.horizonanalyticslabs.com/data-quality Harbor is a format, not a quality mark Harbor makes this work easier. It gives tasks a common structure. Instructions, environments, solutions, and graders can be packaged in a way that is runnable and easier to compare. Without that shared format, reviewing datasets across vendors would be much harder. But Harbor does not decide whether the instruction is complete. It does not know whether the answer is hiding in Git history. It does not prove that the grader tests what the task asks for. Harbor standardizes the package. The audit tells us whether the contents deserve to be trusted. Horizon is building a vendor quality index Buyers should not have to inspect thousands of task folders before finding out whether a dataset works. Horizon is building an index for comparing benchmark and task-data vendors. It will rank the quality of the public datasets we can inspect, show the evidence behind every assessment, and update the result as vendors publish new versions or correct old findings. The assessment will consider whether: - The instruction and grader describe the same task. - The tests cover the work being requested. - The task resists obvious shortcuts and reward hacks. - The environment contains what the model needs without containing the answer. - The reference solution passes reliably. - The vendor maintains the dataset when dependencies or external systems change. The index will start with dataset-level scores. Vendor rankings will aggregate those scores only when there is enough coverage to support the comparison. A preliminary rating based on one sample will be labeled preliminary. A vendor reviewed across several datasets and releases can earn a higher-confidence assessment. This is not meant to freeze vendors into a permanent order. A vendor that fixes a task should receive credit for fixing it. A dataset that changes should not keep a rating earned by an older version. Previous releases will remain visible so buyers can see whether quality is improving. Terminal Bench’s continuous release model is a good example https://www.tbench.ai/news/terminal-bench-4-0 . Its semantic versions distinguish changes that require a new model run, changes that only require regrading, and documentation changes that allow an existing result to be reused. The history stays legible even as the benchmark improves. The point is simple: compare evidence instead of claims. This audit will keep changing We will add datasets, review more tasks, and re-run checks when publishers update their work. Every release will preserve four things: - The dataset version we reviewed. - The checks we were able to run. - The evidence behind each finding. - The date the result was last verified. The ranking is a view of the evidence we have today. It is not a permanent label. The score comes last A benchmark score is the end of a long chain. The instruction has to describe the work. The environment has to contain what the task needs without containing the answer. The reference solution has to work. The grader has to recognize the behavior the instruction asked for. If any one of those fails, the score can still look precise. It is just no longer telling us what we think it is. How we ran this release We scanned 20 datasets containing 5,241 tasks. We identified the largest vendors using their market presence, visibility in the AI data ecosystem, and the scale of their public contributions to Harbor. We then drew random samples from their datasets, inspected 239 tasks more closely, and reviewed 34 one by one. The 29 confirmed tasks come from the audit as a whole, not only the 34-task manual sample. We count a task as broken only when the problem is visible in the current files or reproducible with the published grader or verifier. Anything less remains a review flag. Appendix: every confirmed finding The appendix is deliberately more detailed than the article. A task can appear in several categories but appears only once below. | Vendor / dataset | Task | Confirmed problem | Proof | |---|---|---|---| | Scale AI / SWE-bench Pro | instance ansible ansible-0ea40e09 https://hub.harborframework.com/tasks/scale-ai/instance ansible ansible-0ea40e09d1b35bcb69ff4d9cecf3d0defa4b36e8-v30a923fb5c164d6cd18280c02422f75e611e8fb2 | Fix recovered from history and applied; 16 of 16 tests passed | Real verifier run | | Scale AI / SWE-bench Pro | instance gravitational teleport-db89206d https://hub.harborframework.com/tasks/scale-ai/instance gravitational teleport-db89206db6c2969266e664c7c0fb51b70e958b64 | Fix recovered from history and applied; 4 of 4 tests passed | Real verifier run | | Scale AI / SWE-bench Pro | instance tutao tutanota-fe240cbf https://hub.harborframework.com/tasks/scale-ai/instance tutao tutanota-fe240cbf7f0fdd6744ef7bef8cb61676bcdbb621-vc4e41fd0029957297843cb9dec4a25c7c756f029 | Fix recovered from history and applied; 107 of 107 tests passed | Real verifier run | | Mercor / Apex Agents | world224-sk-task03-dfcbb713 https://hub.harborframework.com/tasks/mercor/world224-sk-task03-dfcbb713 | Positive and negative grading rules require different financial values | Current-file check | | OpenThoughts | cryptographic-protocol-verifier https://hub.harborframework.com/tasks/openthoughts/cryptographic-protocol-verifier | Published solution chooses randomly; grader only checks for PASS or FAIL | Direct grader test | | OpenThoughts | floor-plan-geometry https://hub.harborframework.com/tasks/openthoughts/floor-plan-geometry | Golden answer is 4, while the grader accepts 2, 4, or 6 | Direct grader test | | Terminal Bench Pro | python-pcap-anomaly-detector https://hub.harborframework.com/tasks/terminal-bench-pro/python-pcap-anomaly-detector | Grader requires two output fields the instruction never mentions | Current-file check | | OpenThoughts | word-derangement-mapping https://hub.harborframework.com/tasks/openthoughts/word-derangement-mapping | Replacing the editable dictionary changes the same fake answer from 0 to 1 | Direct grader test | | OpenThoughts | publisher-market-analysis https://hub.harborframework.com/tasks/openthoughts/publisher-market-analysis | NaN bypasses numeric checks and duplicated rows still score 1 | Direct grader test | | DataCurve / Deep SWE | obsidian-linter-auto-table-of-contents https://hub.harborframework.com/tasks/datacurve/obsidian-linter-auto-table-of-contents | Required test names depend on a display name the instruction never provides | Current-file check | | Scale AI / SWE-bench Pro | ansible ansible-8127abbc https://hub.harborframework.com/tasks/scale-ai/instance ansible ansible-8127abbc298cabf04aaa89a478fc5e5e3432a6fc-v30a923fb5c164d6cd18280c02422f75e611e8fb2 | Fix remains in the image; required work is untested; tests expect an unstated signature | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | future-architect vuls-86b60e14 https://hub.harborframework.com/tasks/scale-ai/instance future-architect vuls-86b60e1478e44d28b1aff6b9ac7e95ceb05bc5fc | Fix remains in the image; only the parser is graded; range behavior is ambiguous | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | gravitational teleport-32bcd715 https://hub.harborframework.com/tasks/scale-ai/instance gravitational teleport-32bcd71591c234f0d8b091ec01f1f5cbfdc0f13c-vee9b09fb20c43af7e520f57e9239bbcf46b7113d | Fix remains in the image and the graded run skips the package containing the crash | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | internetarchive openlibrary-3aeec6af https://hub.harborframework.com/tasks/scale-ai/instance internetarchive openlibrary-3aeec6afed9198d734b7ee1293f03ca94ff970e1-v13642507b4fc1f8d234172bf8129942da2c2ca26 | Fix remains in the image; tests miss the described change; gold patch is only a rename | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | internetarchive openlibrary-5de7de19 https://hub.harborframework.com/tasks/scale-ai/instance internetarchive openlibrary-5de7de19211e71b29b2f2ba3b1dff2fe065d660f-v08d8e8889ec945ab821fb156c04c7d2e2810debb | Fix remains in the image and the named function is never tested | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | internetarchive openlibrary-6afdb09d https://hub.harborframework.com/tasks/scale-ai/instance internetarchive openlibrary-6afdb09df692223c3a31df65cfa92f15e5614c01-v08d8e8889ec945ab821fb156c04c7d2e2810debb | The task is otherwise sound, but the finished fix is reachable in the image | Inside-image check | | Scale AI / SWE-bench Pro | internetarchive openlibrary-8a9d9d32 https://hub.harborframework.com/tasks/scale-ai/instance internetarchive openlibrary-8a9d9d323dfcf2a5b4f38d70b1108b030b20ebf3-v13642507b4fc1f8d234172bf8129942da2c2ca26 | Fix remains in the image and tests call private helpers the instruction never names | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | navidrome navidrome-3f2d2469 https://hub.harborframework.com/tasks/scale-ai/instance navidrome navidrome-3f2d24695e9382125dfe5e6d6c8bbeb4a313a4f9 | Fix remains in the image; the graded suite adds constructor calls without assertions | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | navidrome navidrome-5e549255 https://hub.harborframework.com/tasks/scale-ai/instance navidrome navidrome-5e549255201e622c911621a7b770477b1f5a89be | Fix remains in the image and only the storage half of the work is graded | Inside-image and current-file checks | | Scale AI / SWE-bench Pro | navidrome navidrome-812dc209 https://hub.harborframework.com/tasks/scale-ai/instance navidrome navidrome-812dc2090f20ac4f8ac271b6ed95be5889d1a3ca | Fix remains in the image and a one-file partial solution earns full marks | Direct grader and inside-image checks | | Scale AI / SWE-bench Pro | tutao tutanota-d1aa0ece https://hub.harborframework.com/tasks/scale-ai/instance tutao tutanota-d1aa0ecec288bfc800cfb9133b087c4f81ad8b38-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf | Fix remains in the image and one summary line is counted as 107 passing tests | Inside-image and current-file checks | | Terminal Bench / TB2 | git-multibranch https://hub.harborframework.com/tasks/terminal-bench/git-multibranch | Two static files pass without the Git server or hook requested by the task | Current-file check | | Terminal Bench / TB2 | make-mips-interpreter https://hub.harborframework.com/tasks/terminal-bench/make-mips-interpreter | An old frame left in place passes without being tied to the interpreter | Current-file check | | Terminal Bench / TB2 | modernize-scientific-stack https://hub.harborframework.com/tasks/terminal-bench/modernize-scientific-stack | An empty dependency file and two hardcoded numbers pass | Current-file check | | Terminal Bench / TB2 | mteb-retrieve https://hub.harborframework.com/tasks/terminal-bench/mteb-retrieve | Following the instruction returns the wrong document; an unstated argument is required | Current-file check | | Terminal Bench / TB2 | nginx-request-logging https://hub.harborframework.com/tasks/terminal-bench/nginx-request-logging | Three required behaviors are never checked | Current-file check | | Terminal Bench / TB2 | portfolio-optimization https://hub.harborframework.com/tasks/terminal-bench/portfolio-optimization | A NumPy wrapper passes a task that explicitly asks for a C implementation | Current-file check | | Terminal Bench / TB2 | protein-assembly https://hub.harborframework.com/tasks/terminal-bench/protein-assembly | The author’s own solution fails because current external data cannot satisfy the tests | Real verifier run | | Terminal Bench / TB2 | pytorch-model-cli https://hub.harborframework.com/tasks/terminal-bench/pytorch-model-cli | Required behavior is untested and grading depends on an outside MNIST mirror | Current-file check | Appendix: what remains uncertain The audit also contains review flags. They may be real problems, but the evidence is not strong enough to call the tasks broken. - Harvey legal tasks may require more than their instructions state, but a legal expert needs to determine whether those expectations are implicit in the work. - Public examples create a contamination risk for Aider zebra-puzzle tasks, but an answer existing online does not by itself invalidate a task. - BenchFlow’s PDF task reads the current date and installs packages during grading, but we have not observed a correct answer fail. - An exact-output HTML task may reject equivalent formatting, but the intended acceptance rule needs to be confirmed. - A Sakila timing cutoff may be unstable, but we still need a repeatable failure measurement. - An IoT firmware grader requires extraction evidence that the instruction does not state, but that may be an intentional check against hardcoded output. None of these is included in the confirmed counts. Appendix: limits of this release - The initial screening sent almost every real task to review. It was cautious, but not decisive. - Some Snorkel and Mercor graders require paid AI judges we could not run. That is a gap in this audit, not a fault in those datasets. - Each container task was run once unless otherwise stated. A flaky task and a permanently broken task can initially look the same. - Twenty broken tasks in the 34-task manual review is a fact about those 34 tasks. It is not an estimate for the entire market. Findings apply to the dataset versions and task images audited on the stated dates. Publishers can send corrections or updated versions for re-testing. Previous releases will remain available so changes in quality can be tracked over time.