Synthesizing Rollback Systems for Safer Agents Basis detailed a compensator system that gives coding agents transactional semantics when calling stateful APIs, rolling back an agent's operations to the original state if an action cannot be undone or a sequence crashes. Basis built a compensator for Amazon's S3 API, which it says has more than one hundred complex interacting operations, using verified synthesis in which agents both write software and prove it meets desired safety properties. Basis said it intends to extend the work to the rest of AWS and then to every API where a compensator would be useful, citing reported incidents of agents deleting a company's database and posting a personal bank balance to company Slack. Vibe coding has taken the world by storm. But a question remains: how do we use LLMs to produce reliable software? That’s to say, how do we use LLMs to build software that users would actually trust in safety-critical settings? At Basis, we believe that software verification is more important than ever, as it provides necessary security guarantees against increasingly intelligent and capable AI attackers. The software ecosystem is not a fixed set of programs; it is an ever expanding and transforming fabric, and we believe that establishing novel techniques for producing robust software is just as important as securing the systems that we already have. One area of particular importance is building software for agent safety. Agents are a relatively new phenomenon in the history of software, and the software we use to steer and control them is immature and not hardened. If we blindly use agents to generate agent safety software, we run the risk of producing an unreadable pile of slop which does not meaningfully make our systems safer. In this post, we detail our efforts to solve a critical agent safety issue. We approach the issue with verified synthesis, asking agents to both write software and prove that it meets the safety properties we desire. Compensators Software developers increasingly delegate tasks involving stateful APIs to their coding agents. Instead of wrangling together a script of nuanced operations, many instead express their high-level goal to an agent in natural language and leave it to sort out the details. However, natural language is imprecise and agents are fallible, and mistakes can have critical consequences. Users have reported agents deleting a company’s database https://www.theguardian.com/technology/2026/apr/29/claude-ai-deletes-firm-database and posting a personal bank balance to company Slack https://x.com/ShaneMac/status/2107486740491669879 . More generally, any trouble encountered by agents in the middle of a task can leave API state in a complicated intermediate state that is a headache to recover from. Our solution is to mediate agent interactions with an API using a compensator . Try seeing how one works in the figure below. The displayed toy API manipulates a counter; as you send operations from the agent, the compensator will handle interactions with the API and maintain a compensation trace . At any point, you can click the rollback button and the compensator will use the accumulated compensation trace to roll back the API state to its original value. Note that using the set operation causes the compensator to snapshot the current state before modifying it. Mediating accesses with the compensator gives coding agents transactional semantics; either an agents entire workload goes through or none of it does. If an agent tries to take an action that cannot be rolled back or otherwise produces an action sequence that crashes, our system automatically rolls back the state. If the user is dissatisfied with what the agent has done, they can undo its effects with the push of a button. In general, a compensator rolls back operations either by 1 associating operations with corresponding inverse operations as with incremement and decrement or 2 snapshotting the state before the modification and later restoring it as with the set operation. The first approach requires a deep understanding of the relationships between API operations. The second option introduces new failure modes e.g. the snapshot can be too large and crash the compensator and requires understanding how to reach an arbitrary given snapshotted state. In this post, we describe how we developed a compensator for Amazon’s S3 API, which has more than one hundred complex interacting operations. Our intention is to extend this work to the rest of AWS and then to every API for which a compensator would be useful. Prior work has explored agent-native filesystems with undo https://doi.org/10.1145/3830418.3843858 and applying database recovery ideas to agents https://onewill.ai/blog/2026/stealing-50-years-of-database-ideas-for-ai-agents/ . Our focus is synthesizing compensators for opaque, stateful APIs rather than implementing rollback for a filesystem. We structure the remainder of this post around the key insights enabling our work. 1. These APIs are too large to build compensators for by hand. We instead synthesize, or automatically generate, compensators using an agentic loop. 2. Synthesizing compensators from just API documentation produces systems that are sometimes useful but generally buggy. We found that we can significantly improve the soundness of our synthesized systems by verifying them with respect to lightweight “gray box” specifications of the write footprints of API operations. This suggests that API providers can expose relatively inexpensive partial specifications that enable third parties to synthesize more reliable software. 3. Requiring LLMs to synthesize compensators that verify has a tendency to produce overly conservative compensators which refuse most API operations. We recovered functionality by adding a form of positive feedback to the standard counterexample-based synthesis loop. Synthesizing Compensators We automate the process of writing a compensator for an API by generating the compensator using a LLM. One shot generation often results in reasonable code when using a frontier model, but in all of our experiments the generated code has bugs. To reduce the number of bugs in the compensator, we run a feedback loop: 1 given a compensator, an LLM in a fresh context generates a failing test case, and 2 given a failing test and a compensator, an LLM repairs the compensator so that it passes the test. Note that these are not arbitrary failing tests; compensators have a precise specification that we are testing. Any action sequence whose effects we cannot roll back treating a midway abort and rollback as a successful rollback is a valid counterexample. The compensator should either have rejected one of the actions as uncompensable or it should have successfully rolled all of the actions back. Aside: This process is reminiscent of a classic approach to inductive program synthesis called Counterexample Guided Inductive Synthesis CEGIS https://doi.org/10.1145/1168857.1168907 , and it is both sound and complete in the limit when using a sound and complete decision procedure for code and counterexample generation. LLMs, of course, are neither sound nor complete, so our loop offers no formal guarantee of correctness or progress. We found that finding and repairing counterexamples improves the quality of the generated code, but we are left with a question: how can we be confident that we found all the counterexamples? Unfortunately, we cannot be confident that our process generates correct code, because even though they are good bug finders, LLMs are not complete bug finders. Anyone who has used a LLM for code review has experienced this effect: different LLMs find different subsets of valid bugs. Just searching for failing tests is not enough to build confidence, because we can never know if we have missed one. Instead, we need to verify our code. Synthesizing Robust Compensators Verifying any property of a compensator means that we need to state some properties of the API operations. We started by writing down which parts of the API state each operation changes. We call these properties write sets . For the S3 API, a write set is a set of buckets, objects, and attributes that an operation writes to. For example, CreateBucket b has the write set {Bucket b } . Properties like these are much simpler than complete functional descriptions of the behavior of an API operation, so we think that they’re plausible for API engineers to write and maintain as machine-readable documentation, as long as they’re useful. Using these properties, we can prove some theorems about our compensators: 1. When no rollback is performed, the compensator performs no additional writes. 2. When a rollback is performed, the rollback actions write to exactly the same locations as the forward actions. While these properties fall well short of functional correctness, they give us some additional assurance that the code we generate isn’t doing anything too unsafe. The first property tells us that the compensator only touches our API state during a rollback; if no rollback happens, then the state is only changed by the forward actions. The second property tells us that no state changed by a forward action is left unchanged by the rollback. We synthesize our compensators in Dafny https://dafny.org/ , which automatically proves that our code satisfies the properties above by reducing the proof to a SMT query. This style of verification is simpler and more automated than the interactive theorem proving approach used by Lean https://lean-lang.org/ or Rocq https://rocq-prover.org/ , particularly when proving simpler program properties. The LLM does not need to interact with the prover beyond generating code and receiving feedback on its correctness. Introducing verification into our synthesis process gives us much more confidence that we are generating reasonable code, but it has an unfortunate side effect in our use case. We allow compenators to optionally refuse to perform a given operation. This is a necessary escape hatch because some operations for example, sending an email cannot be compensated When we began verifying properties of synthesized compensators, the LLM would produce compensators which refused to perform any operation that modifies state. This solution is technically sound if no actions run, then the initial state is equal to the final state and the compensator is correct , but is obviously undesirable because the agent can’t do anything. We found that the LLM was unlikely to reach for this solution when generating unverified code, but when we began verifying its output, this became common behavior. To deal with this problem, we need to do more than generate compensators that are merely correct. Synthesizing Robust and Featureful Compensators Standard counterexample-driven synthesis loops only provide negative feedback to the proposer, picking at weak points in its proposed candidate. Our verification infrastructure provides even stronger negative signals, scolding the proposer with obtuse error messages whenever its candidate does not pass its check. In the presence of such negativity, it is unsurprising that the proposer becomes overly conservative and refuses any operations that modify state. We recover coverage for these operations by introducing a form of positive feedback which we call an opportunity . An opportunity consists of a test and a single-use compensator such that: 1. the current candidate compensator refuses to complete the test 2. the single-use compensator successfully executes the test and rolls the state back Intuitively, an opportunity is positive feedback that it should also be able to cover the opportunity’s test case, as evidenced by the opportunity’s single-use compensator. Note that because the single-use compensator need only work for one test case, the logic that it implements can be completely custom. One of our key assumptions is the proposer can extract a generalizable improvement to the candidate from any custom single-use compensator covering a blocked test case. We found that incorporating opportunities in our synthesis loop resulted in compensators that were both robust and frequently useful. We evaluated four synthesis variants on 31 held-out S3 tasks. A pass means the workload completed and rollback restored the initial state; a safe abort means the compensator refused an operation and restored the initial state without completing the workload. Errors and state mismatches are separate failure outcomes. With verification and opportunities, completed-and-restored tasks increased from 1 to 13 out of 31 . Across all 60 evaluated checkpoints, that variant had no observed errors or state mismatches in 1,860 task evaluations. Unverified counterexample-only synthesis had 43 errors and one mismatch over the same number of evaluations; unverified synthesis with opportunities had one error and four mismatches. These are repeated evaluations of the same tasks, not independent safety trials. Across models of different strengths, we found that the story of using LLMs to synthesize compensators was largely the same. Current models can produce useful compensators but remain buggy after many rounds of counterexample-driven feedback. Verification alone produces overly conservative compensators; opportunities recover useful behavior while retaining the verified write-set properties. These properties do not prove that rollback restores the correct values, and zero observed failures on this benchmark is not a guarantee of functional correctness. The experiments also assume sequential execution with no concurrent writers and no failures during rollback. Conclusion and Future Work We think this project demonstrates the feasibility of the idea, but there’s a lot of work left to do. We’d like to support more of the AWS API, ideally by making the system more general rather than by doing custom per-API work. There’s substantial work to do on making our harness scale better currently it runs a single agent at a time . The guarantees provided by verification can be expanded and strengthened. There’s more work to build on from the databases community: we should support failures during rollback and in restricted cases support concurrent writers. Overall, we think this work is an interesting demonstration of the value of combining agentic AI and formal methods. If you found this post interesting, check out the other posts on our blog We have several published projects in the AI/formal methods space, with more to come. References - Database deletion incident https://www.theguardian.com/technology/2026/apr/29/claude-ai-deletes-firm-database . The Guardian 2026 . - Personal bank balance posted to company Slack https://x.com/ShaneMac/status/2107486740491669879 . Shane Mac 2026 . - Zhong et al. Don’t Let AI Agents YOLO Your Files: Information and Control in Agent-Native Filesystems https://doi.org/10.1145/3830418.3843858 . SOSP 2026 . - Wan Lim and Will Zhang. Stealing 50 Years of Database Ideas for AI Agents https://onewill.ai/blog/2026/stealing-50-years-of-database-ideas-for-ai-agents/ 2026 . - Solar-Lezama et al. Combinatorial sketching for finite programs https://doi.org/10.1145/1168857.1168907 . ASPLOS 2006 .