{"slug": "new-open-source-benchmark-reveals-reasoning-as-the-biggest-bottleneck-for-ai-in", "title": "New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations", "summary": "CloudOpsBench 1.0, an open-source benchmark built by a team led by Ang Chen at the University of Michigan, found that reasoning rather than knowledge is the main failure mode for AI agents on cloud operations tasks, with Claude Fable 5.1 solving 60 percent of its 45 AWS tasks and GPT-6 Astra solving 53 percent. About 80 percent of failures came from agents misjudging task scope, not checking all relevant services, or stopping too early, while roughly 20 percent were wrong actions, some irreversible. The Apache 2.0-licensed benchmark runs each task identically in the real AWS cloud and in an emulator, with extensions for GCP, Azure, and OCI planned.", "body_md": "# New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations\n\n**CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects.** Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers.\n\nEven the best models fail at many of the tasks. Claude Fable 5.1 solved 60 percent, GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure.\n\nAll tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The [project on GitHub](https://github.com/CloudOpsBench/CloudOpsBench) is licensed under Apache 2.0, and a detailed [failure analysis](https://cloudopsbench.ai/blog/failure-taxonomy) is also available.\n\n```\nAI News Without the Hype – Curated by Humans\n\n\t\t\t\t\tSubscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive \"AI Radar\" frontier report six times a year, full archive access, and access to our comment section.\t\t\t\t\n\n\t\t\t\t\tSubscribe now\n```\n\n[CloudOpsBench](https://cloudopsbench.ai/)", "url": "https://wpnews.pro/news/new-open-source-benchmark-reveals-reasoning-as-the-biggest-bottleneck-for-ai-in", "canonical_source": "https://the-decoder.com/new-open-source-benchmark-reveals-reasoning-as-the-biggest-bottleneck-for-ai-in-cloud-operations/", "published_at": "2026-11-04 16:42:52+00:00", "updated_at": "2026-11-04 16:50:17.385163+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research", "artificial-intelligence", "mlops"], "entities": ["CloudOpsBench", "Ang Chen", "University of Michigan", "Claude Fable 5.1", "GPT-6 Astra", "AWS", "GCP", "Azure"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/new-open-source-benchmark-reveals-reasoning-as-the-biggest-bottleneck-for-ai-in", "markdown": "https://wpnews.pro/news/new-open-source-benchmark-reveals-reasoning-as-the-biggest-bottleneck-for-ai-in.md", "text": "https://wpnews.pro/news/new-open-source-benchmark-reveals-reasoning-as-the-biggest-bottleneck-for-ai-in.txt", "jsonld": "https://wpnews.pro/news/new-open-source-benchmark-reveals-reasoning-as-the-biggest-bottleneck-for-ai-in.jsonld"}}