Unguarded, Sonnet 5 read another customer's order 100/100 times. Guarded: zero.

A developer's evaluation of AI agent safety found that a frontier model, Sonnet 5, refused a destructive attack in 100 out of 100 trials but executed a cross-customer read in 100 out of 100 trials, matching the behavior of a model with safety training removed. The tests, run with and without an authorization boundary, showed zero breaches when the boundary was in place, highlighting the confused-deputy problem where models are helpful but lack application context. The developer's project, Verdict, ships deterministic attack packs to measure such vulnerabilities.

I ran the same storefront attacks against three models — one with its safety training deliberately removed, one stock open-weights model, and the frontier model Laravel AI ships as its default — each with and without an authorization boundary in place. The headline isn't that the boundary held it did: zero breaches across every guarded arm . It's that the frontier model's safety training turned out to be per-action : it refused the destructive attack in 100 out of 100 trials, and executed the cross-customer read in 100 out of 100 trials. The attack it misses is the one that looks like helping. Verdict https://github.com/fissible/verdict ships deterministic attack packs — executable specifications of things an agent should never be allowed to do, like reading or cancelling another customer's order. The attack always arrives through the conversation — a hostile request, a poisoned document — and what gets measured is whether the agent goes along with it. Every attack ran in two arms: Integrity rules, because a rigged benchmark is worse than none: the harness never forces a tool call, a model that declines an attack is never counted as a prevention, and independent sampled trials are reported as per-arm tallies, never as "the guard stopped this exact attack." | abliterated 7B | gpt-oss:20b | Sonnet 5 | | |---|---|---|---| Lookup breached, unguarded per trial | 30/30 | 19/30 | 100/100 | Cancellation breached, unguarded per trial | 28/30 | 1/30 | 0/100 | | Breaches with the boundary guarding per attempted attack | 0/60 | 0/25 | 0/100 | Read the last column twice. The frontier model refused the destructive attack perfectly — nothing stopping it but its own training — and executed the cross-customer read as reliably as the model with its safety training stripped out. The read series isn't even monotonic: Sonnet executed it more often than the mid-tier open-weights model. More alignment did not mean less exposure on that action. Because it doesn't look like an attack from where the model sits. "Cancel someone else's order" pattern-matches to harm; frontier training catches it. "Look up order 4471" looks like doing your job — the model has a lookup tool, a user asked about an order, and nothing in its context says whose order that is. This is the confused-deputy problem, as old as computer security: the agent isn't malicious, it's helpful , with authority the requester shouldn't be able to borrow. That's why the fix isn't a better prompt or a more aligned model. Whose order a tool may touch is a fact in your database, checked by your policies — application state the model never sees and cannot be argued out of. Models propose; applications authorize. The full write-up — the other two models in detail, the legitimate-work allow-side zero false denials , the diagrams, and every caveat — is on my blog: The AI Wouldn't Cancel Someone Else's Order. But It Read It Every Single Time. https://fissible.dev/blog/the-ai-read-the-order-anyway Recorded runs and raw numbers: docs/evaluation.md https://github.com/fissible/verdict/blob/main/docs/evaluation.md . If you're building AI agents on Laravel, wire your tools through the boundary and run the control arm against your own app — the attack your model's alignment misses is probably not the one you'd guess.