AutomationBench-AA

Artificial Analysis launched AutomationBench-AA, an independent leaderboard for Zapier's AutomationBench, testing AI agents on 657 real SaaS workflow automation tasks across 40 simulated apps. Claude Fable 5 leads with 48.6% objective completion, but every model tested violated business rules, with Gemini 3.5 Flash achieving the best ratio of objectives per guardrail violation.

All articles /articles July 6, 2026 Announcing AutomationBench-AA Announcing AutomationBench-AA, our independent leaderboard for Zapier’s AutomationBench, testing whether AI agents can automate real SaaS workflows while adhering to business rules. We partnered with Zapier to run AutomationBench-AA on their private benchmark subset. This benchmark is a complex agentic workflow automation test across simulated SaaS applications. Models must complete 657 tasks spanning Finance, HR, Marketing, Operations, Sales, and Support, working across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot. Unlike Zapier’s hosted leaderboard, the headline score for AutomationBench-AA shows the share of objectives a model completes without violating any guardrails. Claude Fable 5 and Opus 4.8 from Anthropic lead with scores of 48.6% and 48.5%, followed by Google DeepMind's Gemini 3.5 Flash at 42.6% and OpenAI's GPT-5.5 xhigh at 42.1%. With Anthropic’s new classifier, Fable 5 fell back to Opus on ~18% of tasks. Key elements of AutomationBench: ➤ Real workflow patterns, simulated environments: Tasks are drawn from real workflow patterns on Zapier and run in simulated SaaS environments, where a single task may span a range of applications like CRM, email, calendar, and messaging platforms. ➤ Autonomous API discovery: Models interact with each app through REST APIs, discovering the endpoints they need through structured tool calls and navigating environments with irrelevant and sometimes misleading records. ➤ Objectives and guardrails: Models are scored against nearly 12,000 assertions Zapier built to test that the model completed the task correctly in full. Each assertion is classified as either an objective the agent must achieve, or a guardrail that already passes initially and must not be broken. ➤ Programmatic environment grading: Tasks are graded solely on whether the correct data ended up in the right systems, with deterministic checks against the environment. Each task runs once with a 50-turn cap. Key results for AutomationBench-AA: ➤ Claude Fable 5 max leads at 48.6% but falls back to Opus 4.8 in ~18% of tasks. It completes 73% of task objectives, with the fallback behavior likely explaining the limited uplift compared to Opus. ➤ Every model breaks business rules: Guardrail violations range from 0.46 per task Gemini 3.5 Flash to 1.26 Qwen3.7 Plus . Gemini 3.5 Flash completes 15.0 objectives per guardrail violation, the best ratio of any model, ahead of Claude Opus 4.8 max, 13.5 . ➤ Gemini 3.5 Flash performs well for its price: At 42.6% and $0.49 per task, it effectively matches GPT-5.5 xhigh, 42.1%, $1.32 per task at ~37% of the cost. ➤ GLM-5.2 max from Z.ai is the leading open weights model at 27.8%. This places the open weights frontier ~10 points behind Gemini 3.1 Pro Preview, and with substantially higher guardrail violations per task. ➤ Finance workflow tasks are the most difficult to automate today: across the models we evaluated at launch, agents complete roughly half the proportion of objectives on Finance tasks, compared to Support and Operations tasks. We would like to thank Zapier and the benchmark authors for their great work developing this benchmark for important SaaS workflows, and appreciate their collaboration in launching AutomationBench on Artificial Analysis AutomationBench-AA evaluates models on 657 workflow automation tasks developed by Zapier from real workflow patterns on its platform. Models orchestrate work across 40 simulated SaaS app environments via REST APIs, and every task is graded programmatically on the final state of those systems. While completing objectives, agents must also avoid breaking guardrails that represent business rules. Every model we evaluated at launch triggers guardrail violations, and violation-adjusted efficiency separates the leaders: Gemini 3.5 Flash completes 15.0 objectives per violation and Claude Opus 4.8 max 13.5. Task difficulty varies significantly by business domain. Finance workflows are the hardest to automate: across all models, agents complete around one third of Finance objectives, roughly half the rate of Support and Operations ~60% . Cost per task spans more than an order of magnitude across the models we evaluated, from <5 cents for DeepSeek V4, Gemini 3.1 Flash-Lite, and Qwen3.7 Plus to nearly $1.50 for models like Claude Opus 4.8 max . The leaders are not always more expensive than peers: Gemini 3.5 Flash delivers its third-place 42.6% at $0.49 per task. Working styles differ sharply on AutomationBench. GPT-5.5 xhigh takes a more action-intensive approach, averaging 49 tool calls across 25 turns per task, while Claude Opus 4.8 max is more deliberate: 35 tool calls packed into just 14 turns, with fewer guardrail violations 0.55 vs 0.66 per task . Grok 4.3 high takes the fewest turns 13 , but does not perform as strongly as models which persist longer to complete tasks, consistent with declaring tasks complete prematurely rather than finishing them efficiently. For more information see full results of AutomationBench-AA on our website: https://artificialanalysis.ai/evaluations/automationbench-aa https://artificialanalysis.ai/evaluations/automationbench-aa Zapier leaderboard: https://zapier.com/benchmarks https://zapier.com/benchmarks ArXiv paper: https://arxiv.org/abs/2604.18934 https://arxiv.org/abs/2604.18934 AutomationBench on GitHub: https://github.com/zapier/AutomationBench https://github.com/zapier/AutomationBench Read the latest Claude Sonnet 5: strong agentic performance at a higher cost per task Claude Sonnet 5 achieves 53 on the Artificial Analysis Intelligence Index, but without promotional pricing will cost more per task than Opus 4.8 June 30, 2026 Measuring time per task in AA-Briefcase Agentic knowledge work can take frontier models over 20 minutes per task, as measured in AA-Briefcase, our new benchmark June 24, 2026 Announcing the Artificial Analysis Speech to Speech Index Announcing the Artificial Analysis Speech to Speech Index, our new synthesis metric for native Speech to Speech model quality, comprising of Big Bench Audio, Full Duplex Bench, and 𝜏-Voice June 23, 2026