{"slug": "we-found-defects-in-37-of-deepswe-s-113-tasks", "title": "We found defects in 37 of DeepSWE's 113 tasks", "summary": "An audit of DeepSWE v1.1 found defects or ambiguous requirements in 37 of the benchmark's 113 tasks (32.7%), based on a review of all 372 recorded failures for the Opus 5, Sol and Fable 5 models. The defects included hidden tests that broke the build, such as Go naming collisions that caused 18 false failures across five tasks, and assertions that rejected valid output, including four Sol submissions that failed only because a warning message omitted the word \"invalid\". DeepSWE v1.1 is cited in OpenAI's GPT-6 Astra launch table, which reports scores of 74.1% for Astra, 73.7% for Opus 5, 72.7% for Sol and 69.9% for Fable 5.", "body_md": "[DeepSWE v1.1](https://deepswe.datacurve.ai/) is used to compare some of the most capable AI models available. It appears in OpenAI’s [GPT-6 Astra launch table](https://openai.com/index/gpt-6-astra/) and Anthropic’s [Fable 5.1 system card](https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20%26%20Claude%20Mythos%205.1%20System%20Card.pdf). OpenAI’s table reports scores of **74.1% for Astra, 73.7% for Opus 5, 72.7% for Sol and 69.9% for Fable 5**. Differences of a percentage point or less appear to separate several of these models.\n\nYet when we reviewed all 372 recorded failures for Opus 5, Sol and Fable 5, we found defects or ambiguous requirements in **37 of DeepSWE’s 113 tasks (32.7%)**. These included hidden tests that broke the build, assertions that rejected valid output, and tests that enforced requirements the instructions left unclear.[1](#fn1)\n\nOne task asks the model to [add typed variables](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/anko-typed-variable-bindings/instruction.md) to a scripting language implemented in Go. The prompt encourages testing, and the agent added a test for the new feature.\n\nAfter the agent submitted its solution, the benchmark injected hidden tests into the same Go package to evaluate it. Those tests were unavailable to the agent while it worked. The prompt did not describe this injection step or reserve any test names:\n\n```\n// Agent's test, written before submission\nfunc TestTypedBindingsDeclarations(t *testing.T) { ... }\n\n// Hidden test, injected during evaluation\nfunc TestTypedBindingsDeclarations(t *testing.T) { ... }\n```\n\nDeclaring two functions with the same name in the same package is a compilation error, so adding the hidden tests broke the build. The agent had no opportunity to see or resolve the collision; the benchmark recorded a failure without testing the feature.\n\nRenaming the hidden tests allowed all three affected submissions to pass. This was part of a wider pattern: Go naming collisions caused 18 false failures across five tasks.\n\nOther failures came down to a single word. [One task](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/testem-bail-on-test-failure/instruction.md) required invalid configuration values to produce a warning through `npmlog`, with the prefix `bail_on_test_failure`, and then fall back to `false`. The submission did this:\n\n```\nlog.warn('bail_on_test_failure',\n  'Expected true, false, or a positive integer; defaulting to false.');\nreturn false;\n```\n\nBut the test also required the warning to contain “invalid”:\n\n```\nexpect(warnStub.firstCall.args[1]).to.match(/invalid/i);\n```\n\nThe prompt specified the logging library, prefix and fallback, but did not prescribe the message’s wording. The warning explained the accepted values and the fallback. Four Sol submissions failed solely because “invalid” was missing. We classed this as a defect: keeping the warning and fallback checks while removing the word requirement made all four pass.\n\nA [SQL formatting task](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/sqlfmt-create-table-ddl-formatting/instruction.md) shows the same gap in a different form. Eight submissions failed tests that required `IN (` to become `IN(`, but the prompt did not settle whether that space should remain. We classed this as an ambiguity rather than a defect: the test is not wrong, but a formatting test needs an expected style, and the instructions must tell the agent what that style is.\n\nOther examples:\n\n| Example | Occurrences | Type | \n|---|---|---|\n| A valid Markdown link, `[My Page](<My Page>)` , was rejected in favor of the invalid`[My Page](My Page)` .[`obsidian-linter-link-format-conversion`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/obsidian-linter-link-format-conversion/instruction.md) | 6 | Defect | \n| All 41 feature tests passed, but the scorer treated them as missing because it expected a different display label. [`obsidian-linter-auto-table-of-contents`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/obsidian-linter-auto-table-of-contents/instruction.md) | 4 | Defect | \n| CSS tests rejected `auto` where they expected the equivalent`auto auto` .[`csstree-shorthand-expansion-compression`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/csstree-shorthand-expansion-compression/instruction.md) | 5 | Defect | \n| Test fixtures omitted paths, then rejected the missing-path warnings the prompt required. [`helm-array-merge-strategies`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/helm-array-merge-strategies/instruction.md) | 5 | Defect | \n| A test rejected an enabled toolbar because `aria-disabled` was absent rather than explicitly set to`\"false\"` .[`quill-shared-toolbar-focus`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/quill-shared-toolbar-focus/instruction.md) | 4 | Defect | \n| A returned timeout was `5` seconds; the test expected`5000` milliseconds. The return unit was unspecified.[`kombu-virtual-queue-dead-lettering`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/kombu-virtual-queue-dead-lettering/instruction.md) | 7 | Ambiguity | \n| Tests required integer milliseconds, rejecting fractional timings even though the prompt prescribed no integer type. [`prometheus-transactional-reload-status`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/prometheus-transactional-reload-status/instruction.md) | 4 | Ambiguity | \n| An environment flag set to `\"0\"` could mean false or a truthy, nonempty string; the prompt left this unclear.[`abs-module-cache-flags`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/abs-module-cache-flags/instruction.md) | 2 | Ambiguity | \n| A folded HTTP header retained a tab; the test expected a space. The prompt did not specify that normalization. [`httpx-multipart-response-parsing`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/httpx-multipart-response-parsing/instruction.md) | 1 | Ambiguity | \n| At zero display width, the test required an invisible hyperlink open/close pair; empty output also satisfied the stated constraints. [`termenv-preserve-ansi-resets`](https://github.com/datacurve-ai/deep-swe/blob/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea/tasks/termenv-preserve-ansi-resets/instruction.md) | 3 | Ambiguity | \n\nTallied by task:\n\n| Finding | Tasks | Share | \n|---|---|---|\n| Confirmed evaluator defect only | 10 | 8.8% | \n| Ambiguous requirements only | 21 | 18.6% | \n| Both defect and ambiguity | 6 | 5.3% | \n| No issue identified | 76 | 67.3% | \n| All tasks | 113 | 100% | \n\nFixing the confirmed defects and rerunning the original submissions turned 67 failures into passes without changing the submitted code. This raised the three models’ measured pass rates by 4.22 to 6.19 percentage points:[2](#fn2)\n\n| Model | Archived score | **After verified corrections** | Verified change | If all ambiguities also pass† | \n|---|---|---|---|---|\n| Opus 5 | 73.65% | **78.38%** | +4.73 pp | 84.91% | \n| Sol | 72.67% | **76.89%** | +4.22 pp | 86.89% | \n| Fable 5 | 69.72% | **75.92%** | +6.19 pp | 81.42% | \n\n† Hypothetical: adds all 98 ambiguous failures to the verified passes. The requirements must be clarified and the submissions rerun to test this assumption. pp = percentage points.\n\n**The larger problem is that these evaluation problems remain in a benchmark cited as evidence of frontier capability.** A score cannot distinguish an implementation error from a scoring error or an unstated requirement. Optimize against such scores without reading the failures, and you train models to satisfy the evaluator rather than solve the problem.\n\nAs coding benchmarks approach saturation, the remaining gap to 100% becomes harder to interpret. Before counting another percentage point as progress, we should check what the point was awarded for.\n\n1. \nCodex assisted with the review of all 372 recorded failures; a human expert reviewed the 180 flagged cases. We tested evaluator corrections on the archived submissions without generating new solutions, using published task images.\n2. \nThe archived runs number 444 for Opus 5, 450 for Sol and 436 for Fable 5. Corrections preserve those populations and count only unchanged submissions that passed all required tests after repair. The final column assumes all 98 ambiguous failures also become passes; this has not been tested. Fable’s archived baseline is 69.72%, rather than the launch table’s 69.9%. We did not audit Astra or Fable 5.1.", "url": "https://wpnews.pro/news/we-found-defects-in-37-of-deepswe-s-113-tasks", "canonical_source": "https://www.scrimdata.com/blog/deep-swe-benchmark-defects-ambiguities", "published_at": "2026-09-19 12:56:19+00:00", "updated_at": "2026-09-19 13:25:13.104908+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "developer-tools"], "entities": ["DeepSWE", "DataCurve", "OpenAI", "GPT-6 Astra", "Anthropic", "Claude Fable 5.1", "Opus 5", "Sol"], "alternates": {"html": "https://wpnews.pro/news/we-found-defects-in-37-of-deepswe-s-113-tasks", "markdown": "https://wpnews.pro/news/we-found-defects-in-37-of-deepswe-s-113-tasks.md", "text": "https://wpnews.pro/news/we-found-defects-in-37-of-deepswe-s-113-tasks.txt", "jsonld": "https://wpnews.pro/news/we-found-defects-in-37-of-deepswe-s-113-tasks.jsonld"}}