Via news.microsoft.com
New open-source framework reveals a troubling gap between AI agents that can solve a task once and those that can do it consistently Microsoft has released ThinkingBox, an open-source sandbox framework designed to test whether AI agents can actually be trusted to handle real business tasks.
The tool, detailed in a Microsoft Command Line blog post on August 19, 2026, by Principal Machine Learning Engineer Liang-Chun Tsai, takes a fundamentally different approach to evaluating AI agents. Instead of grading them on what they say they did, ThinkingBox checks what they actually changed in a database.
The discovery-reliability gap #
ThinkingBox comes paired with a benchmark called ThinkingBox-Bench, and the initial results are sobering. Microsoft tested 12 different proprietary and open-weight models across 507 specific tasks spanning five business domains. Each task was run through 20 separate trials, creating a rigorous statistical picture of how these agents perform under repeated conditions.
The best-performing model managed a 65.36% pass@1 rate, meaning it completed a task correctly on its first try roughly two-thirds of the time. That sounds reasonable until you see the pass^20 number: 25.25%.
Pass^20 measures whether a model can succeed across all 20 trials of the same task. Microsoft calls this the “discovery-reliability gap,” and it’s the central finding of the entire project.
Why checking the database matters #
Traditional methods for evaluating AI agents tend to rely on transcript analysis, essentially reading the agent’s work like a teacher grading an essay. ThinkingBox sidesteps this entirely by verifying the actual back-end database records after each task. The framework creates a clean, isolated environment for every trial, so each test starts from the same baseline. Then it runs executable assertions against the final state of the system, checking whether the data reflects a successful completion.
Microsoft’s results showed that numerous unsuccessful trials involved what appeared to be valid actions or clean terminations. The agent didn’t crash or throw errors. It just quietly did the wrong thing, or did part of the right thing and stopped. These are exactly the kinds of failures that transcript-based evaluation would miss.
Where this fits in Microsoft’s AI strategy #
ThinkingBox is part of a broader push by Microsoft toward what the company frames as agent observability and governance. A 25.25% pass^20 rate from the strongest available model means that even the best AI agents fail to deliver consistent results across repeated attempts at the same task roughly three-quarters of the time.
The framework and its benchmark are now available on GitHub at github.com/microsoft/thinkingbox. The open-source release is a deliberate choice: Microsoft appears to want the evaluation methodology to become an industry standard rather than a proprietary advantage.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our