The Accio team's state-based benchmark reports Qwen3.8-Max completing 56 tasks through Accio, 47 through OpenClaw and 53 through Pi; Alibaba calls it the strongest overall open-weight result.
By RuntimeWire Staff · Published
Primary source: Qwen / Alibaba
Why it matters #
Agent vendors need tests that inspect completed business state instead of polished transcripts. The Accio team's benchmark advances that approach, while Alibaba's self-reported two-pass open-weight margin, harness variance and lack of immutable public task-level bundles make auditability as important as leaderboard order.
Yukun Lian, Sicong Xie and ten colleagues on Alibaba International's Accio team built Commerce Agent Bench to test whether AI agents can finish commercial work inside stateful replicas of business software. Alibaba's Qwen3.8-Max, part of the Qwen3.8 model family, completed 56 of 107 tasks through Accio, 47 through OpenClaw and 53 through Pi. Alibaba's official Qwen account described that as the strongest overall performance among open-weight models.
The repository's three published result tables show Qwen3.8-Max ahead of DeepSeek V4 Pro by two aggregate passes, while closed model Claude Opus 5 finished 35 passes ahead of Qwen. Those figures come from Alibaba's own benchmark, so the ranking is best read as a published result rather than an independent audit.
Lian and Xie are also listed as contributors to Business Arena, another Accio project that puts agents in charge of a simulated cross-border shop over a longer operating period. Their work follows a consistent thesis: models should be judged through the state they leave behind after acting, with business outcomes carrying more weight than a convincing transcript.
A benchmark that checks the state
Commerce Agent Bench contains 107 tasks: 53 command-line tasks, 28 browser tasks, 16 file and document tasks, and 10 API or MCP tasks. They cover work such as publishing products, booking freight, editing storefronts, analyzing suppliers, researching the public web and producing spreadsheets.
Each task starts in a fresh container. The agent receives a business request, a workspace and access to local software replicas. According to the benchmark methodology, task graders, rubrics, private seeds and mock implementations remain outside the agent-visible tree. After the agent stops, a host-side verifier reads the final state of the mock services and any files produced. A task passes only when every required check passes, according to the repository.
That construction addresses a familiar weakness in agent demonstrations. A model can describe a plausible series of actions, report that it finished and still leave the database, storefront or document unchanged. Commerce Agent Bench checks the saved result.
Alibaba's Accio team modeled commerce and logistics surfaces including product publishing, freight booking, storefront administration and payment operations. The suite uses 14 offline software replicas, according to the benchmark methodology. The project is published under Apache 2.0 for its code and CC BY 4.0 for its task data, with commercial use permitted under those licenses.
Qwen's two-pass lead comes from one harness
The repository's published results show that Qwen3.8-Max passed 56 of 107 tasks through Accio, 47 through OpenClaw and 53 through Pi, for 156 passes across 321 harness-task combinations. That equals a simple aggregate completion rate of 48.6%.
The same result tables record 53 passes for DeepSeek V4 Pro with Pi, 47 with OpenClaw and 54 with Accio. Its 154 passes equal 48.0% across the three harnesses. Qwen tied DeepSeek in Pi and OpenClaw, then finished two tasks ahead in Accio. That difference is the basis for Alibaba's strongest-overall wording.
Claude Opus 5 led all three harnesses, according to the published result tables, passing 65 tasks with Pi, 60 with OpenClaw and 66 with Accio. Its 191 passes across 321 harness-task combinations produce a 59.5% aggregate rate, 10.9 percentage points above Qwen's 48.6%.
The scores also show how far commercial agents remain from unattended operation. Even the leading model failed 41 of 107 tasks in its best harness. For a merchant trusting an agent with listings, freight, payments or customer records, a 61.7% pass rate still demands supervision and transaction-level checks.
The harness can move the result
The spread between Qwen's three scores is a warning against treating a model name as a complete agent specification. Qwen finished nine more tasks in Accio than in OpenClaw. Claude Opus 5 moved by six tasks between its weakest and strongest harnesses.
The repository's Qwen integration documentation says Qwen returns its reasoning trace in reasoning_content
, which must be replayed in subsequent assistant messages. It says both published Qwen3.8-Max rows used that protocol and warns that results produced without it are not comparable with the published figures. The benchmark methodology also says provider routing, model snapshots, prompt adapters, retry policies and judge endpoints can affect the measured result.
The benchmark materials identify Accio as a reference-only harness whose results cannot be reproduced from the repository checkout, while OpenClaw is the shipped, rerunnable harness. The materials also say the raw task-level result bundles behind the published board are not stored in Git or made available through immutable public archives with checksums. Full reproduction of the published leaderboard therefore remains unclear even though the per-harness tables are public.
Publishing task-level result bundles with immutable identifiers and checksums would make the two-pass open-weight margin easier to audit independently.
Private evaluations widen the benchmark's remit
The repository documentation says the maintainers offer evaluations for pre-release and internal model checkpoints, including private tests. It also invites outside contributors to propose task domains and mock environments, with reviewed replicas eligible for inclusion in later benchmark versions.
Those programs can broaden the workflows covered by the suite, while private evaluations remain separate from the evidence available for public leaderboard claims. The distinction matters most when only a handful of passes separate two models.
The benchmark evaluates agents; it does not compete as a shopping service. Alibaba's Accio team has released a concrete set of workflows where agents can fail visibly, through an unsaved listing, a missed booking or an incorrect document. Qwen's narrow open-weight lead gives Alibaba a marketing line. The 107 tasks give other developers a place to find out whether an agent actually did the job.