{"slug": "deepswe-results-ten-coding-agents-on-one-model-2026", "title": "DeepSWE results: ten coding agents on one model (2026)", "summary": "AgentField's open-source coding agent CodeAF solved 62 of 113 DeepSWE tasks (54.9%) in senior-dev mode, placing first among ten harnesses that all ran the same model, deepseek/deepseek-v4-flash-0731, through OpenRouter. CodeAF also posted the lowest cost per solved issue at 39.9 cents (22 cents per task), ahead of mini-swe-agent at 56 of 113 (49.6%) and 1.9x that cost, while Claude Code and the DeepSeek Harness each solved 16 of 113. The same mode reached 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3, with per-task records published for all 113 tasks per harness.", "body_md": "*I work on [codeaf](https://github.com/Agent-Field/CodeAF) at AgentField, one of the harnesses compared here. This post first appeared on [agentfield.ai](https://agentfield.ai/benchmarks/deepswe).*\n\nCodeAF, the open-source coding agent by AgentField, solved 62 of 113 DeepSWE tasks (54.9%) in senior-dev mode, first of ten harnesses in AgentField's DeepSWE comparison, where every harness ran the same model (`deepseek/deepseek-v4-flash-0731` through OpenRouter). It also had the lowest cost per solved issue: 39.9 cents, at 22 cents a task. Every number is in [`arms.csv`](https://github.com/Agent-Field/CodeAF/tree/d049b13406dc9179f2265608bafca3255f3c5f3a/docs/benchmarks/deepswe).\n\n| Result | CodeAF (senior-dev mode) solved 62 of 113 (54.9%), first of ten harnesses in this comparison | \n| Model, for every harness | `deepseek/deepseek-v4-flash-0731` through OpenRouter | \n| Cost | 22 cents a task, 39.9 cents per solved issue (the lowest of the ten) | \n| Next best | mini-swe-agent, 56 of 113 (49.6%), 1.9x CodeAF's cost per solved issue | \n| With newer models | the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3 | \n\nEvery row ran the full DeepSWE set on the same model. \"Cost per solved issue\" is each harness's spend divided by tasks solved, shown as a multiple of CodeAF's 39.9 cents. \"Mean time\" is mean agent wall time per task.\n\n| Harness | Solved (of 113) | Cost per task | Cost per solved issue | Mean time | \n|---|---|---|---|---|\n| **CodeAF (senior-dev)** | **62, 54.9%** | **22 cents** | **1x** | 54 min | \n| mini-swe-agent | 56, 49.6% | 38 cents | 1.9x | 44 min | \n| Codex CLI | 51, 45.1% | 37 cents | 2.1x | 46 min | \n| Pi | 42, 37.2% | 35 cents | 2.4x | 52 min | \n| Oh My Pi (omp) | 31, 27.4% | 50 cents | 4.5x | 49 min | \n| OpenCode | 30, 26.5% | 50 cents | 4.8x | 48 min | \n| Kilo Code | 30, 26.5% | 48 cents | 4.6x | 54 min | \n| Claude Code | 16, 14.2% | 19 cents | 3.4x | 32 min | \n| DeepSeek Harness | 16, 14.2% | 150 cents | 26.6x | 94 min | \n| Muse Code | 3, 2.7% | 12 cents | 11.3x | 16 min | \n\n30 of 113 is 26.5%. The method page prints 26.6% for OpenCode and Kilo Code, rounded from the stored reward rate of 0.2655; the task count is the same. In `arms.csv` the CodeAF row is named `senior-dev`, because the binary had an earlier name on the day of the run. At a shell, the same mode is `codeaf senior-dev`; in a conversation it is `/senior-dev` ([senior-dev docs](https://agentfield.ai/docs/codeaf/senior-dev)).\n\nThe [per-task records](https://github.com/Agent-Field/CodeAF/tree/9b4344aaf56048c995cb9caa33369449cf137832/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks), one CSV per harness with 113 rows each, show where the lead came from:\n\n| Benchmark | full DeepSWE set, 113 tasks | \n| Model | `deepseek/deepseek-v4-flash-0731` through OpenRouter | \n| Verifiers | official DeepSWE at `0b9fabb` | \n| Budget | 3 hours per task | \n| Isolation | four shards per harness, a dedicated OpenRouter key per harness | \n\nHow the columns are derived, from the [method page](https://github.com/Agent-Field/CodeAF/tree/d049b13406dc9179f2265608bafca3255f3c5f3a/docs/benchmarks/deepswe):\n\n| Harness | Version tested | Release date (UTC) | \n|---|---|---|\n| CodeAF | first stable release containing senior-dev mode: [v0.5.0](https://github.com/Agent-Field/CodeAF/releases/tag/v0.5.0) (integration commit`dd0fcc654` ,[#1488](https://github.com/Agent-Field/CodeAF/pull/1488) ) | 2026-09-30 | \n| mini-swe-agent | [2.4.6](https://github.com/SWE-agent/mini-swe-agent/releases/tag/v2.4.6) | 2026-07-23 | \n| Codex CLI | [0.154.0](https://github.com/openai/codex/releases/tag/rust-v0.154.0) | 2026-09-09 | \n| Pi | [0.85.1](https://github.com/earendil-works/pi/releases/tag/v0.85.1) | 2026-09-05 | \n| Oh My Pi | [18.1.17](https://github.com/can1357/oh-my-pi/releases/tag/v18.1.17) | 2026-09-10 | \n| OpenCode | [1.18.30](https://github.com/anomalyco/opencode/releases/tag/v1.18.30) | 2026-09-09 | \n| Kilo Code | [7.6.2](https://github.com/Kilo-Org/kilocode/releases/tag/v7.6.2) | 2026-09-10 | \n| Claude Code | [2.1.267](https://github.com/anthropics/claude-code/releases/tag/v2.1.267) | 2026-09-09 | \n| DeepSeek Harness ( `@deepseek-ai/dsh` ) | [0.1.5-rc.1](https://github.com/deepseek-ai/deepseek-harness/releases/tag/dsh-v0.1.5-rc.1) | 2026-09-10 | \n| Muse Code | 1.1.1-R2514.1 | not independently established | \n\n[`versions.csv`](https://github.com/Agent-Field/CodeAF/tree/d049b13406dc9179f2265608bafca3255f3c5f3a/docs/benchmarks/deepswe) holds full source revisions and release timestamps.\n\n[DeepSWE](https://deepswe.datacurve.ai/), published by Datacurve, is \"a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks\": 113 tasks across TypeScript, Go, Python, JavaScript and Rust, each graded by a verifier that \"accepts any solution whose observable behavior is correct\" ([DeepSWE README](https://github.com/datacurve-ai/deep-swe)). [SWE-bench Verified](https://www.swebench.com/verified.html) is \"a human-validated subset of 500 SWE-bench instances.\"\n\nAgentField's comparison holds the model fixed and changes the harness, so it measures what the agent around the model adds. Every harness was graded by the same verifiers on the same tasks.\n\nWith newer models, the same mode solved 77.9% on DeepSeek V4.1 Flash and 69.0% on Kimi K3. That is 88 of 113 solved (exact 95% CI 69.1% to 85.1%) with DeepSeek V4.1 Flash and 78 of 113 (59.6% to 77.4%) with Kimi K3.\n\nThe scripts that run and grade DeepSWE tasks are in [`bench/deepswe/`](https://github.com/Agent-Field/CodeAF/tree/dev/bench/deepswe) in the CodeAF repository. One task through CodeAF:\n\n```\nbench/deepswe/run.sh <task-id> <model-id> <seed-tag>\n```\n\nIt needs a linux/amd64 `codeaf` binary in `CODEAF_BIN`, a provider key, and the DeepSWE task corpus, which Datacurve distributes as a [gated dataset](https://huggingface.co/datasets/datacurve/deep-swe). The `compare-*` scripts beside it install a pinned comparator, freeze its container image and run it. The [per-task records](https://github.com/Agent-Field/CodeAF/tree/9b4344aaf56048c995cb9caa33369449cf137832/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks) of this campaign are public, so every table on this page can be recounted from them.\n\nCodeAF was first of ten harnesses in AgentField's DeepSWE comparison, solving 62 of 113 tasks (54.9%) with every harness on DeepSeek V4 Flash. The comparison holds the model fixed and ranks the harness around it.\n\nWhen the model is held fixed (DeepSeek V4 Flash) and the harness changes, CodeAF solved the most DeepSWE tasks of the ten harnesses in AgentField's comparison, 62 of 113 (54.9%), ahead of mini-swe-agent (56) and Codex CLI (51).\n\nIn AgentField's DeepSWE comparison on DeepSeek V4 Flash, CodeAF had the lowest cost per solved issue, 39.9 cents, ahead of mini-swe-agent (1.9x), Codex CLI (2.1x) and Claude Code (3.4x).\n\nOn DeepSeek V4 Flash, in AgentField's DeepSWE comparison, CodeAF solved 62 of 113 tasks and Claude Code 2.1.267 solved 16. Both ran DeepSeek V4 Flash, not Claude, so the result measures the harness on that model, not Claude Code with Claude models.\n\nYes. The run and grading scripts are in `bench/deepswe/` in the CodeAF repository, and the per-task records of all ten harnesses are public, so every count here can be checked. The DeepSWE task corpus is a gated dataset from Datacurve.\n\n`arms.csv` and `versions.csv` beside it: every number on this page, checked 2026-11-21.`bench/deepswe/`\nFor how CodeAF compares beyond this benchmark, see [CodeAF vs Claude Code](https://agentfield.ai/compare/codeaf-vs-claude-code) and [CodeAF vs Codex CLI](https://agentfield.ai/compare/codeaf-vs-codex-cli). For the 3-hour, unattended way senior-dev works, see [a coding agent that runs for hours](https://agentfield.ai/guides/long-running-coding-agent), and for the terminal side, [terminal coding agents](https://agentfield.ai/guides/terminal-coding-agent).\n\nLast updated 2026-11-21. Every figure checked against `arms.csv` and `versions.csv` at the method page; harness versions are those of the 2026-09-10 to 12 campaign.", "url": "https://wpnews.pro/news/deepswe-results-ten-coding-agents-on-one-model-2026", "canonical_source": "https://dev.to/santoshkumarradha/deepswe-results-ten-coding-agents-on-one-model-2026-3moh", "published_at": "2026-11-21 20:06:55+00:00", "updated_at": "2026-11-21 20:12:03.781134+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "developer-tools"], "entities": ["AgentField", "CodeAF", "DeepSWE", "mini-swe-agent", "Codex CLI", "Claude Code", "OpenRouter", "DeepSeek"], "also_reported_by": [], "takeaway": "On the full 113-task DeepSWE set with deepseek-v4-flash-0731, CodeAF's senior-dev mode solved 54.9% at 39.9 cents per solved issue, versus 49.6% for mini-swe-agent at 1.9x the cost and 14.2% for Claude Code.", "recommendation": {"advice": "Benchmark coding harnesses on the same model and compare cost per solved issue, not just solve rate; the spread ran from 54.9% to 2.7% on identical tasks.", "playbook": "coding-agents", "playbook_url": "https://wpnews.pro/playbooks/coding-agents"}, "alternates": {"html": "https://wpnews.pro/news/deepswe-results-ten-coding-agents-on-one-model-2026", "markdown": "https://wpnews.pro/news/deepswe-results-ten-coding-agents-on-one-model-2026.md", "text": "https://wpnews.pro/news/deepswe-results-ten-coding-agents-on-one-model-2026.txt", "jsonld": "https://wpnews.pro/news/deepswe-results-ten-coding-agents-on-one-model-2026.jsonld"}}