Startup wants to make AI agents safe for DevOps duty

Vera Platforms, a startup founded by University of Michigan researchers, is building three products — vera-gym, vera-bench and vera-agents — to make AI agents reliable enough for DevOps work, CEO Ang Chen told The Register. Chen said Anthropic's Claude and OpenAI's Codex score only about a 50 percent pass rate on the company's CloudOpsBench test, and that when asked to decommission AWS resources the models only acted in the default us-east-1 region, leaving resources in other regions untouched. Vera-agents, trained in the vera-gym simulated infrastructure environment, is slated to be available in Q1 2027.

Startup wants to make AI agents safe for DevOps duty The Register https://www.theregister.com Infrastructure automation needs something better than prompt and pray Vera Platforms, a startup founded by researchers at the University of Michigan, believes there's a way to make agents trustworthy enough to handle DevOps. DevOps, which focuses on how software gets deployed and operated, is a particularly unforgiving affair. While AI agents have some leeway to make mistakes and correct errors as they generate code and work through a prompt, it's an entirely different matter when they make changes to real infrastructure running production workloads. "Live infrastructure is unforgiving," Ang Chen, CEO of Vera Platforms https://vera-platforms.com and an associate professor at the University of Michigan, told The Register. "An incorrect change can take down a service, open a security hole or erase data. Mistakes are often irreversible, and there’s no safe way to experiment: running code against production infra just to see what happens isn’t an option." For the past few years, Chen and his colleagues have been working https://arxiv.org/abs/2506.12270 on a way to benchmark https://cloudopsbench.ai/ agentic DevOps and a gym – a simulation environment – to help AI models get better at DevOps tasks. "From the research that we did, we've figured out various weaknesses of leading models," said Chen. "For instance, Anthropic https://www.machinebrief.com/glossary/anthropic 's Claude https://www.machinebrief.com/glossary/claude , although it's very strong with Python generation, it just has some fundamental limitations that it can't get right when it comes to DevOps." An example of that, Chen said, is that when they asked Claude or OpenAI https://www.machinebrief.com/glossary/openai 's Codex to decommission resources in the AWS cloud, the models didn't look beyond the default region. "The default region is us-east-1," he explained. "There are various regions in AWS, but Claude doesn't have this awareness that the cloud resources could exist in multiple regions, so it only decommissioned part of the resource." The fundamental reason, he said, is that these models have seen a lot of public code, but a lot less operational tracing. Vera Platforms aims to remedy the situation with three products: vera-gym, a playground environment that emulates production infrastructure; vera-bench, a test for measuring where AI agents fumble DevOps tasks; and vera-agents, AI agents that have been trained in vera-gym and are ready to be deployed for DevOps tasks. The CloudOpsBench test being released builds upon work done in 2024 https://proceedings.neurips.cc/paper files/paper/2024/hash/f26b29298ae8acd94bd7e839688e329b-Abstract-Datasets and Benchmarks Track.html that was presented at NeurIPS. According to Chen, Claude and Codex only score about a 50 percent pass rate. "It's a very, very hard set of tasks," he said. The test covers tasks like deleting decommissioned S3 buckets https://github.com/CloudOpsBench/CloudOpsBench/blob/v1.0/tasks/aws/delete-decommissioned-s3-buckets/instruction.md , repairing a compliance event pipeline https://github.com/CloudOpsBench/CloudOpsBench/blob/v1.0/tasks/aws/repair-evidence-event-routing/instruction.md , and restoring DynamoDB write autoscaling https://github.com/CloudOpsBench/CloudOpsBench/blob/v1.0/tasks/aws/restore-dynamodb-write-autoscaling/instruction.md . "The tasks are authored by senior infra operators with many years of experience, with rigorous peer review," said Chen. This benchmark https://www.machinebrief.com/glossary/benchmark has the advantage of not requiring an actual cloud deployment. "We have a high fidelity simulator that mimics Amazon behavior," said Chen. "So the rollout, the performance of the agent on the real cloud and the performance of the agent in our emulator is exactly the same. So for every API, we have this comparison and the agent sees the same thing." Vera-agents, Chen said, should be available in Q1 2027. In their academic work, the researchers describe cloud infrastructure as "critical but tedious work." With a bit more training https://www.machinebrief.com/glossary/training , maybe AI agents can take over and keep things boring. ® Get AI news in your inbox Daily digest of what matters in AI.