A new benchmark built from 26,314 real coding sessions tests whether agents can finish the change a user asked for. We explain how it scores, what the gap between partial and full credit means, and which results hold up across judges and harnesses.
This explains reporting by
arXiv 2610.35138: Co-Design Bench, Multimodal Collaborative Coding Evaluation for Agentic Models (submitted October 26, 2026). Read the original first:
https://arxiv.org/abs/2610.35138
In one minute #
- Co-Design Bench has 273 tasks taken from 37,463 deduplicated requests in 26,314 real coding sessions, each with hidden expert acceptance criteria (2,331 in all).
- The best of ten models, GPT-6 Astra, passes 69.9% of criteria on an average task but meets every criterion on only 26.4% of tasks. Claude Opus 5 is second at 18.3%, and Grok 4.6 last at 6.6%.
- Repository edits are much harder than single edits: GPT-6 Astra fully finishes 41.0% of multimodal edits but 17.9% of repository edits.
- Every model was run inside Claude Code as the harness, with 3 hours and 120 turns per task. An LLM judge grades, and a second LLM tries to refute each pass.
- The paper reports no human agreement figure for the judges and no cost per task. Treat the ranking as solid and the exact percentages as estimates.
What the benchmark asks #
Most coding benchmarks hand a model an isolated bug and a test suite. Co-Design Bench starts from what real users asked for. The authors took 39,325 request records from 26,314 coding sessions, removed exact and near duplicates (word overlap above 0.80), and rebuilt each app so it would build, launch in a container and render in a browser. Apps that needed private APIs or logins were dropped.
They then ranked requests by difficulty, how checkable they are, and signs the user struggled, such as rephrasing or rolling back. Experts rewrote the best ones and wrote hidden acceptance criteria. The result is 273 tasks in two tracks: 100 multimodal edits (a change to an app, sometimes with an image, audio, video or document attached) and 173 repository edits (coordinated changes across a codebase).
How it scores, and why two numbers #
Each task has a list of hidden criteria, 2,331 across the benchmark. A judge agent reads the code diff, renders the app and drives it in a browser, then marks each criterion pass, partial or fail. A second model then tries to refute every pass and partial, looking for stubs, hard-coded results, features nobody can reach, or work that does not match the request. Visual claims need a screenshot and interaction claims need a browser log, or the grading stops.
That gives two numbers. The strict score is the share of criteria a model passed. The full-pass rate is the share of tasks where it passed every criterion. The first measures progress. The second measures whether a user would get what they asked for without going back in. The gap between them, 69.9% against 26.4% for the top model, is the headline: agents do most of a job far more often than they finish it.
The full table #
Strict score, then full-pass rate, across all 273 tasks, from the paper:
- GPT-6 Astra: 0.699, 26.4%
- Claude Opus 5: 0.661, 18.3%
- Qwen3.8-Max: 0.653, 16.5%
- Qwen3.8-27B: 0.644, 15.0%
- Kimi K3: 0.629, 15.8%
- GPT-5.6 Sol: 0.611, 14.3%
- Claude Opus 4.8: 0.581, 10.3%
- GLM-5.3 Flash: 0.573, 10.6%
- GLM-5.2: 0.567, 9.5%
- Grok 4.6: 0.511, 6.6%
Two things stand out. Qwen3.8-27B, a model small enough to self-host, sits within a few points of models many times its size. And the newest frontier models are not on the list: there is no Opus 5.5, Fable 5.1 or GPT-6.1 Sol, so this is a snapshot of what was tested, not of today's best.
Where agents fall down #
Repository work is the hard part. On multimodal edits the full-pass rates run from 6.0% to 41.0%; on repository edits, from 5.8% to 17.9%. Within repository edits, every model scored lower on changes that touch three or four kinds of change than on one or two (0.591 against 0.654 strict, pooled). The share of tasks where a model scored zero rose from 0.9% to 5.8%.
Attachments hurt too. Pooled across models, multimodal edits with no attachment were fully passed 37.0% of the time, with an image 10.0%. The paper calls these comparisons descriptive, not controlled, so read them as patterns rather than causes.
How much to trust the numbers #
The authors re-scored two models with three other models as judges and found graded scores moved by at most 0.053, and the order of those two models never flipped. Two frontend experts reviewed 200 graded outputs, but the paper gives no agreement figure for that review. So the ranking looks stable, while any single percentage could be a few points off.
All ten models ran inside Claude Code as the harness, with prompts, tools and containers held fixed, a 3-hour limit and 120 turns. The authors also ran two models in Codex, opencode and qwen-code; The Decoder reports Qwen3.8-27B scored 75% or 56% on larger overhauls depending on the harness. A model's score here is partly a score of how well it fits Claude Code's tools.
Each task is one fixed request with no chance to ask a question or get feedback, which is not how people work with agents. That likely makes full-pass rates lower than a real session with a human in the loop.
What it means for your team #
- Plan for review and cleanup on most agent changes, especially ones that span several files or kinds of change.
- Write acceptance criteria before you hand an agent a ticket. The benchmark's whole method is hidden criteria, and the same list makes your own review faster.
- Split multi-part changes into single-purpose tasks. Agents scored higher on edits with one or two kinds of change.
- When you compare models, test them in the harness you actually use.
Who is affected #
| Case | Status |
|---|---|
| Teams sending multi-file or multi-step changes to coding agents | Full-pass rates on repository edits were 17.9% at best. |
| Teams choosing between models | The ranking held under other judges, but the newest models were not tested. |
What to do #
- Write down acceptance criteria before assigning a ticket to an agent.
- Keep a human review step on every agent change that touches more than one area.
- Test candidate models in your own harness before switching.
What is still unknown #
- Whose sessions the 26,314 came from. The paper does not say where they were collected; The Decoder lists Alibaba among the authors' affiliations.
- How often the judges agree with human experts. Two experts reviewed 200 outputs, with no agreement figure reported.
- Cost and time per task. The paper reports token use, about 1.24 million tokens per multimodal case for GPT-6 Astra, but no dollars or minutes.
- Whether the tasks and criteria are released. The arXiv page links a project page but no dataset.
- How newer models such as Opus 5.5 or GPT-6.1 Sol would score.
Sources #
- arXiv 2610.35138: Co-Design Bench, Multimodal Collaborative Coding Evaluation for Agentic Models (submitted October 26, 2026) — the original report
- The Decoder: AI coding agents still need human cleanup on nearly three out of four tasks (November 8, 2026)
- [Co-Design Bench project page](https://codesignbench.github.io/)
[AI News Report](https://theainewsreport.com/)· every headline, every morning.