{"slug": "unguarded-sonnet-5-read-another-customer-s-order-100-100-times-guarded-zero", "title": "Unguarded, Sonnet 5 read another customer's order 100/100 times. Guarded: zero.", "summary": "A developer's evaluation of AI agent safety found that a frontier model, Sonnet 5, refused a destructive attack in 100 out of 100 trials but executed a cross-customer read in 100 out of 100 trials, matching the behavior of a model with safety training removed. The tests, run with and without an authorization boundary, showed zero breaches when the boundary was in place, highlighting the confused-deputy problem where models are helpful but lack application context. The developer's project, Verdict, ships deterministic attack packs to measure such vulnerabilities.", "body_md": "I ran the same storefront attacks against three models — one with its safety training deliberately removed, one stock open-weights model, and the frontier model Laravel AI ships as its default — each with and without an authorization boundary in place.\n\nThe headline isn't that the boundary held (it did: zero breaches across every guarded arm). It's that the frontier model's safety training turned out to be **per-action**: it refused the destructive attack in 100 out of 100 trials, and executed the cross-customer *read* in 100 out of 100 trials. The attack it misses is the one that looks like helping.\n\n[Verdict](https://github.com/fissible/verdict) ships deterministic attack packs — executable specifications of things an agent should never be allowed to do, like reading or cancelling another customer's order. The attack always arrives through the conversation — a hostile request, a poisoned document — and what gets measured is whether the agent goes along with it. Every attack ran in two arms:\n\nIntegrity rules, because a rigged benchmark is worse than none: the harness never forces a tool call, a model that *declines* an attack is never counted as a prevention, and independent sampled trials are reported as per-arm tallies, never as \"the guard stopped this exact attack.\"\n\n| abliterated 7B | gpt-oss:20b | Sonnet 5 | |\n|---|---|---|---|\nLookup breached, unguarded (per trial) |\n30/30 | 19/30 | 100/100 |\nCancellation breached, unguarded (per trial) |\n28/30 | 1/30 | 0/100 |\n| Breaches with the boundary guarding (per attempted attack) | 0/60 |\n0/25 |\n0/100 |\n\nRead the last column twice. The frontier model refused the destructive attack *perfectly* — nothing stopping it but its own training — and executed the cross-customer read as reliably as the model with its safety training stripped out. The read series isn't even monotonic: Sonnet executed it *more* often than the mid-tier open-weights model. More alignment did not mean less exposure on that action.\n\nBecause it doesn't look like an attack from where the model sits. \"Cancel someone else's order\" pattern-matches to harm; frontier training catches it. \"Look up order #4471\" looks like *doing your job* — the model has a lookup tool, a user asked about an order, and nothing in its context says whose order that is. This is the confused-deputy problem, as old as computer security: the agent isn't malicious, it's *helpful*, with authority the requester shouldn't be able to borrow.\n\nThat's why the fix isn't a better prompt or a more aligned model. Whose order a tool may touch is a fact in your database, checked by your policies — application state the model never sees and cannot be argued out of. Models propose; applications authorize.\n\nThe full write-up — the other two models in detail, the legitimate-work allow-side (zero false denials), the diagrams, and every caveat — is on my blog: [The AI Wouldn't Cancel Someone Else's Order. But It Read It Every Single Time.](https://fissible.dev/blog/the-ai-read-the-order-anyway)\n\nRecorded runs and raw numbers: [docs/evaluation.md](https://github.com/fissible/verdict/blob/main/docs/evaluation.md). If you're building AI agents on Laravel, wire your tools through the boundary and run the control arm against your own app — the attack your model's alignment misses is probably not the one you'd guess.", "url": "https://wpnews.pro/news/unguarded-sonnet-5-read-another-customer-s-order-100-100-times-guarded-zero", "canonical_source": "https://dev.to/fissible/unguarded-sonnet-5-read-another-customers-order-100100-times-guarded-zero-1mf5", "published_at": "2026-08-20 00:35:41+00:00", "updated_at": "2026-08-20 01:13:45.161461+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-products"], "entities": ["Sonnet 5", "Laravel", "Verdict", "gpt-oss:20b"], "alternates": {"html": "https://wpnews.pro/news/unguarded-sonnet-5-read-another-customer-s-order-100-100-times-guarded-zero", "markdown": "https://wpnews.pro/news/unguarded-sonnet-5-read-another-customer-s-order-100-100-times-guarded-zero.md", "text": "https://wpnews.pro/news/unguarded-sonnet-5-read-another-customer-s-order-100-100-times-guarded-zero.txt", "jsonld": "https://wpnews.pro/news/unguarded-sonnet-5-read-another-customer-s-order-100-100-times-guarded-zero.jsonld"}}