We found defects in 37 of DeepSWE's 113 tasks

An audit of DeepSWE v1.1 found defects or ambiguous requirements in 37 of the benchmark's 113 tasks (32.7%), based on a review of all 372 recorded failures for the Opus 5, Sol and Fable 5 models. The defects included hidden tests that broke the build, such as Go naming collisions that caused 18 false failures across five tasks, and assertions that rejected valid output, including four Sol submissions that failed only because a warning message omitted the word "invalid". DeepSWE v1.1 is cited in OpenAI's GPT-6 Astra launch table, which reports scores of 74.1% for Astra, 73.7% for Opus 5, 72.7% for Sol and 69.9% for Fable 5.

DeepSWE v1.1 https://deepswe.datacurve.ai/ is used to compare some of the most capable AI models available. It appears in OpenAI’s GPT-6 Astra launch table https://openai.com/index/gpt-6-astra/ and Anthropic’s Fable 5.1 system card https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20%26%20Claude%20Mythos%205.1%20System%20Card.pdf . OpenAI’s table reports scores of 74.1% for Astra, 73.7% for Opus 5, 72.7% for Sol and 69.9% for Fable 5 . Differences of a percentage point or less appear to separate several of these models. Yet when we reviewed all 372 recorded failures for Opus 5, Sol and Fable 5, we found defects or ambiguous requirements in 37 of DeepSWE’s 113 tasks 32.7% . These included hidden tests that broke the build, assertions that rejected valid output, and tests that enforced requirements the instructions left unclear. 1 fn1 One task asks the model to add typed variables https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/anko-typed-variable-bindings/instruction.md to a scripting language implemented in Go. The prompt encourages testing, and the agent added a test for the new feature. After the agent submitted its solution, the benchmark injected hidden tests into the same Go package to evaluate it. Those tests were unavailable to the agent while it worked. The prompt did not describe this injection step or reserve any test names: // Agent's test, written before submission func TestTypedBindingsDeclarations t testing.T { ... } // Hidden test, injected during evaluation func TestTypedBindingsDeclarations t testing.T { ... } Declaring two functions with the same name in the same package is a compilation error, so adding the hidden tests broke the build. The agent had no opportunity to see or resolve the collision; the benchmark recorded a failure without testing the feature. Renaming the hidden tests allowed all three affected submissions to pass. This was part of a wider pattern: Go naming collisions caused 18 false failures across five tasks. Other failures came down to a single word. One task https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/testem-bail-on-test-failure/instruction.md required invalid configuration values to produce a warning through npmlog , with the prefix bail on test failure , and then fall back to false . The submission did this: log.warn 'bail on test failure', 'Expected true, false, or a positive integer; defaulting to false.' ; return false; But the test also required the warning to contain “invalid”: expect warnStub.firstCall.args 1 .to.match /invalid/i ; The prompt specified the logging library, prefix and fallback, but did not prescribe the message’s wording. The warning explained the accepted values and the fallback. Four Sol submissions failed solely because “invalid” was missing. We classed this as a defect: keeping the warning and fallback checks while removing the word requirement made all four pass. A SQL formatting task https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/sqlfmt-create-table-ddl-formatting/instruction.md shows the same gap in a different form. Eight submissions failed tests that required IN to become IN , but the prompt did not settle whether that space should remain. We classed this as an ambiguity rather than a defect: the test is not wrong, but a formatting test needs an expected style, and the instructions must tell the agent what that style is. Other examples: | Example | Occurrences | Type | |---|---|---| | A valid Markdown link, My Page <My Page , was rejected in favor of the invalid My Page My Page . obsidian-linter-link-format-conversion https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/obsidian-linter-link-format-conversion/instruction.md | 6 | Defect | | All 41 feature tests passed, but the scorer treated them as missing because it expected a different display label. obsidian-linter-auto-table-of-contents https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/obsidian-linter-auto-table-of-contents/instruction.md | 4 | Defect | | CSS tests rejected auto where they expected the equivalent auto auto . csstree-shorthand-expansion-compression https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/csstree-shorthand-expansion-compression/instruction.md | 5 | Defect | | Test fixtures omitted paths, then rejected the missing-path warnings the prompt required. helm-array-merge-strategies https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/helm-array-merge-strategies/instruction.md | 5 | Defect | | A test rejected an enabled toolbar because aria-disabled was absent rather than explicitly set to "false" . quill-shared-toolbar-focus https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/quill-shared-toolbar-focus/instruction.md | 4 | Defect | | A returned timeout was 5 seconds; the test expected 5000 milliseconds. The return unit was unspecified. kombu-virtual-queue-dead-lettering https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/kombu-virtual-queue-dead-lettering/instruction.md | 7 | Ambiguity | | Tests required integer milliseconds, rejecting fractional timings even though the prompt prescribed no integer type. prometheus-transactional-reload-status https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/prometheus-transactional-reload-status/instruction.md | 4 | Ambiguity | | An environment flag set to "0" could mean false or a truthy, nonempty string; the prompt left this unclear. abs-module-cache-flags https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/abs-module-cache-flags/instruction.md | 2 | Ambiguity | | A folded HTTP header retained a tab; the test expected a space. The prompt did not specify that normalization. httpx-multipart-response-parsing https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/httpx-multipart-response-parsing/instruction.md | 1 | Ambiguity | | At zero display width, the test required an invisible hyperlink open/close pair; empty output also satisfied the stated constraints. termenv-preserve-ansi-resets https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/termenv-preserve-ansi-resets/instruction.md | 3 | Ambiguity | Tallied by task: | Finding | Tasks | Share | |---|---|---| | Confirmed evaluator defect only | 10 | 8.8% | | Ambiguous requirements only | 21 | 18.6% | | Both defect and ambiguity | 6 | 5.3% | | No issue identified | 76 | 67.3% | | All tasks | 113 | 100% | Fixing the confirmed defects and rerunning the original submissions turned 67 failures into passes without changing the submitted code. This raised the three models’ measured pass rates by 4.22 to 6.19 percentage points: 2 fn2 | Model | Archived score | After verified corrections | Verified change | If all ambiguities also pass† | |---|---|---|---|---| | Opus 5 | 73.65% | 78.38% | +4.73 pp | 84.91% | | Sol | 72.67% | 76.89% | +4.22 pp | 86.89% | | Fable 5 | 69.72% | 75.92% | +6.19 pp | 81.42% | † Hypothetical: adds all 98 ambiguous failures to the verified passes. The requirements must be clarified and the submissions rerun to test this assumption. pp = percentage points. The larger problem is that these evaluation problems remain in a benchmark cited as evidence of frontier capability. A score cannot distinguish an implementation error from a scoring error or an unstated requirement. Optimize against such scores without reading the failures, and you train models to satisfy the evaluator rather than solve the problem. As coding benchmarks approach saturation, the remaining gap to 100% becomes harder to interpret. Before counting another percentage point as progress, we should check what the point was awarded for. 1. Codex assisted with the review of all 372 recorded failures; a human expert reviewed the 180 flagged cases. We tested evaluator corrections on the archived submissions without generating new solutions, using published task images. 2. The archived runs number 444 for Opus 5, 450 for Sol and 436 for Fable 5. Corrections preserve those populations and count only unchanged submissions that passed all required tests after repair. The final column assumes all 98 ambiguous failures also become passes; this has not been tested. Fable’s archived baseline is 69.72%, rather than the launch table’s 69.9%. We did not audit Astra or Fable 5.1.