The Register Infrastructure automation needs something better than prompt and pray
Vera Platforms, a startup founded by researchers at the University of Michigan, believes there's a way to make agents trustworthy enough to handle DevOps.
DevOps, which focuses on how software gets deployed and operated, is a particularly unforgiving affair. While AI agents have some leeway to make mistakes and correct errors as they generate code and work through a prompt, it's an entirely different matter when they make changes to real infrastructure running production workloads.
"Live infrastructure is unforgiving," Ang Chen, CEO of Vera Platforms and an associate professor at the University of Michigan, told The Register. "An incorrect change can take down a service, open a security hole or erase data. Mistakes are often irreversible, and there’s no safe way to experiment: running code against production infra just to see what happens isn’t an option."
For the past few years, Chen and his colleagues have been working on a way to benchmark agentic DevOps and a gym – a simulation environment – to help AI models get better at DevOps tasks. "From the research that we did, we've figured out various weaknesses of leading models," said Chen. "For instance, [Anthropic's] Claude, although it's very strong with Python generation, it just has some fundamental limitations that it can't get right when it comes to DevOps."
An example of that, Chen said, is that when they asked Claude or OpenAI's Codex to decommission resources in the AWS cloud, the models didn't look beyond the default region.
"The default region is us-east-1," he explained. "There are various regions in AWS, but Claude doesn't have this awareness that the cloud resources could exist in multiple regions, so it only decommissioned part of the resource."
The fundamental reason, he said, is that these models have seen a lot of public code, but a lot less operational tracing.
Vera Platforms aims to remedy the situation with three products: vera-gym, a playground environment that emulates production infrastructure; vera-bench, a test for measuring where AI agents fumble DevOps tasks; and vera-agents, AI agents that have been trained in vera-gym and are ready to be deployed for DevOps tasks.
The CloudOpsBench test being released builds upon work done in 2024 that was presented at NeurIPS. According to Chen, Claude and Codex only score about a 50 percent pass rate. "It's a very, very hard set of tasks," he said.
The test covers tasks like deleting decommissioned S3 buckets, repairing a compliance event pipeline, and restoring DynamoDB write autoscaling. "The tasks are authored by senior infra operators with many years of experience, with rigorous peer review," said Chen.
This benchmark has the advantage of not requiring an actual cloud deployment.
"We have a high fidelity simulator that mimics Amazon behavior," said Chen. "So the rollout, the performance of the agent on the real cloud and the performance of the agent in our emulator is exactly the same. So for every API, we have this comparison and the agent sees the same thing."
Vera-agents, Chen said, should be available in Q1 2027.
In their academic work, the researchers describe cloud infrastructure as "critical but tedious work." With a bit more training, maybe AI agents can take over and keep things boring. ®
Get AI news in your inbox
Daily digest of what matters in AI.