Tool-using LLM agents often violate their own policies, raising questions about AI governance. Deterministic gates offer a partial solution by reducing silent failures.
Tool-using large language models (LLMs) are supposed to follow rules, right? Yet, notably, these AI systems can violate the very policies they're tasked with enforcing. This paradox leaves industries relying on AI in a precarious position, trust the machine or double-check every result?
Understanding the Silent Failures #
In policy-permissive environments, a tool might execute any well-formed call even if it leads to forbidden state transitions. The numbers tell a different story: on a budget airline domain, 78% of observed failures were silent wrong-state failures. That means no tool error was apparent, yet the process went wrong. The architecture matters more than the parameter count here.
These failures aren't just noise. They're consistent across various seeds, suggesting a systemic issue rather than random errors. Imagine a booking being cancelled or a claim processed without verification. Such silent errors could have significant real-world impacts.
Deterministic Gates: A Partial Solution #
Enter deterministic, read-only pre-execution gates. These gates inspect proposed calls and current states before allowing a write, effectively acting as a safety net. When applied to the gpt-4o-mini model, these gates raised full-benchmark success from 29.6% to 42.0%. That's a 12.4 percentage point jump, substantial in any technical field.
This intervention shows promise. It reproduces its lift across a distinct 15-seed set, adding 12.3 percentage points to success. The key here's where the gates fire: in 26 out of 50 tasks, success improved by 19.2 percentage points. Meanwhile, tasks where the gates didn't fire showed no marked improvement.
Looking Forward: The Frontier Models #
Let's talk about frontier models like gpt-5.2. Even with advanced reasoning, these models still attempt policy-violating writes. Notably, the same suite of deterministic gates boosts success from 61.2% to 71.6%. That's a 10.4 percentage point increase. It's clear these gates help in policy-permissive contexts, but offer little where tools are self-enforcing.
Can we rely on deterministic gates as a silver bullet? Probably not. They don't guarantee task success, but they do help prevent a known class of silent policy-violating writes. This is a bounded evaluation, offering reliability without complete assurance.
So, what's the takeaway? If your industry relies on AI, you might want to consider similar interventions. Silent failures aren't just a technical glitch, they're a potential business risk. The question remains, how much should we trust machines when their oversight can be so easily bypassed?
Get AI news in your inbox
Daily digest of what matters in AI.