Researchers from the Qwen team and collaborators have released Co-Design Bench, a benchmark that asks a practical question: can a coding agent turn a real user's multimodal request into a complete, working application revision? Their answer for today's models is "partly." Across ten model endpoints under one shared agentic harness, the strongest, gpt-6-astra, passes an average of 69.9% of acceptance criteria per task, yet fully satisfies only 26.4% of tasks (paper page on Hugging Face; arXiv 2610.35138).
The gap between those two numbers is the point. Anyone who has asked an agent to change an existing app knows the feeling: most of the request lands, and one broken piece means you still cannot ship it. This benchmark measures that gap.
TL;DR #
| Question | Answer |
|---|---|
| What is it? | A benchmark of 273 real-user app-revision tasks with 2,331 hidden expert-written acceptance criteria |
| Where do tasks come from? | 37,463 deduplicated queries across 26,314 real user coding sessions |
| Tracks | Multimodal Editing (100 tasks), Repository Editing (173 tasks) |
| Models tested | Ten endpoints under a shared agentic harness |
| Best result | gpt-6-astra: 69.9% of criteria per task, 26.4% of tasks fully satisfied |
| Qwen3.8-27B | 64.4% overall strict completion (4th), 64.5% on Repository Editing (2nd), 77.9% on the text-only Multimodal Editing subset |
| Grading | Reference-free judge-then-refute with source and browser evidence |
| Released | Project page, paper, code and evaluation tools, dataset |
What the benchmark actually asks #
Each task starts with a runnable application and a request to revise it. The agent must understand the intent, make the changes, and satisfy every acceptance criterion. The authors describe requests such as reproducing distinct text animations from a video, correcting a dashboard from a receipt and updating its dependent totals, or coordinating layout and interaction changes across an application.
Tasks span three capability dimensions:
- Grounded Understanding: reading what the user points at, such as an annotated screenshot.
- UI Transformation: changing how the interface looks and behaves.
- Functional Implementation: making the underlying logic actually work.
Many requests combine several of these in one go. That is the realistic part: real users rarely ask for one isolated change.
Two tracks
Multimodal Editing (100 tasks) covers localized revisions grounded in annotated screenshots, with optional image, audio, video or document inputs. Repository Editing (173 tasks) covers coordinated changes across components, routes, state, data flow and content. A separate subset called ME-96, with 96 Multimodal Editing tasks and 521 acceptance criteria, is a different scope from the full 273-task paper evaluation, so do not mix numbers between them.
How it grades without a gold answer #
Most coding benchmarks compare output with a reference implementation or run unit tests. Co-Design Bench uses neither. There is no gold implementation and no target screenshot. Instead the authors built a judge-then-refute protocol, shown in their Figure 3:
- Sandbox execution. The output is built (a Vite build or static HTML) and served, then rendered in Chrome.
- Evidence extraction. The system collects the code diff, the page's DOM text, a screenshot, and Playwright action logs.
- Primary evaluation. A judge model reads the request and the hidden expert rubric and issues Pass, Partial or Fail per criterion, with trace evidence.
- Adversarial verification. A separate LLM call challenges positive verdicts, checking for stubs and hardcoding, data-flip tests, entry-to-UI reachability, and semantic criteria.
- Claim audits. Visual claims require screenshots, interaction claims require recorded browser actions, and a violation aborts the claim.
- Scoring. Pass counts 1, Partial 0.5, Fail 0, aggregated into a raw score.
For low-scoring cases with a startable baseline, a parallel solvability probe attributes failures to model limitations, environment blockage, mixed causes or defects in the starting artifact. The figure caption states that raw scores and denominators never change, and environment-adjusted scores are kept separately as supplementary analysis. In practice this means a model cannot earn credit by merely describing a change or by producing code that never renders: each positive verdict has to be backed by a screenshot, a DOM state or a recorded browser action that a reader could inspect afterwards.
The design directly targets a known weakness of LLM-graded coding: an agent can fake a feature with a hardcoded string or an unreachable button. The refute step exists to catch that. It still depends on LLM judges, which carry their own error, so the scores are best read as comparative.
The numbers, and what they say #
The abstract and the authors' summary give the main results:
- Partial versus complete. The best endpoint passes 69.9% of criteria per task on average but fully satisfies only 26.4% of tasks. Criterion-level progress is high, while "done" is rare.
- Complexity hurts. All ten endpoints achieve lower criterion completion on repository requests that span three or four edit types than on those that span one or two. Coordinated, multi-part changes are where agents slip.
- A small open model is competitive. Qwen3.8-27B reaches 64.4% overall strict completion, fourth overall, and 64.5% on Repository Editing, second on that track in this evaluation. It also leads the text-only Multimodal Editing subset at 77.9%.
A fair reading needs care here. The paper is from the Qwen organization, and Qwen3.8-27B is the authors' own family, so the model's strong showing deserves independent replication. The authors themselves caution that "Rankings vary with the editing setting; the small repository-track score differences do not establish a general model advantage." We only have the numbers the authors chose to highlight on the paper page; the full per-endpoint table is in the paper and the project page, and we have not independently rerun the benchmark.
For model background, see our coverage of the Qwen3.8 Max open-weights release and the GPT-6 Sol and Luna launch. Note that the paper lists the endpoint as "gpt-6-astra," so that is the name we use.
Why the 26.4% number matters for builders #
If you evaluate agents by percentage of tests passed, a 70% score sounds nearly ready. A task-level view says otherwise: roughly three of four real revisions still need a human to finish. That matters in three places. Review cost. An agent that is 90% right on every task can still leave a bug in each. The cost is the reviewer's time, which grows with how many separate places a change touches.
Task sizing. The drop on three-to-four-edit-type repository requests suggests splitting big requests into smaller steps, each with its own acceptance check, which fits the loop-based workflow in our loop engineering guide.
Benchmark literacy. Our guide on how to read AI benchmarks makes the same point: look at the unit of success. The Terminal-Bench audit claims show how fragile task definitions can be, and Real-SWE's private-codebase benchmark attacks contamination from another angle. Co-Design Bench's contribution is real-user, multimodal, running-app tasks.
How it compares with other agent evaluations #
| Aspect | Co-Design Bench | Typical issue-fix benchmark |
|---|---|---|
| Task source | Real user requests from coding sessions | Curated issues from open-source repos |
| Input | Screenshots, audio, video, documents, text | Mostly text |
| Grading | Reference-free, browser and source evidence | Hidden unit tests |
| Output measured | Per-criterion pass rate and full-task success | Usually pass or fail per task |
The reference-free grading is what lets it handle UI work where there is no single right diff. The trade-off is reliance on LLM judges and a browser environment, which can themselves fail; the solvability probe is the authors' answer to that.
How to try it #
The authors link everything from the project page: demos of real cases, the evaluation tools, and the results. The code lives in a GitHub repository and the dataset is on Hugging Face. A sensible way to use it today:
- Run your own agent harness on the Multimodal Editing track first, since it has fewer tasks.
- Report both numbers, mean criterion pass rate and fully solved tasks, so the gap is visible.
- Compare repository tasks by how many edit types they span, since that is where every tested endpoint degraded.
- Treat judge verdicts as noisy and spot-check with the saved screenshots and browser logs.
The Qwen team is also behind related agent work such as OpenGCUA, an open stack for generalist computer-use agents, which shares the theme of grounding agents in real interfaces.
Limits to keep in mind #
- Authors are model builders. Qwen's own model appears in the results. Independent reruns will say more.
- Ten endpoints is a snapshot. Harness choices affect results, and the paper's numbers are tied to one shared harness.
- LLM judges. The judge-then-refute design reduces fake passes but cannot remove judge error.
- Scope. The tasks target application revision, mostly web-style apps built with tools like Vite, so results may not transfer to backend-heavy or embedded work.
- Early. The paper was published on October 26 and surfaced on Hugging Face's daily papers on October 30, so expect revisions and community scrutiny.
Related reading #
Figures reflect the Co-Design Bench paper page and project page as of October 30, 2026 and may change in later versions.