{"slug": "i-built-a-system-that-attacks-its-own-ai-agents-every-night-here-s-what-it-found", "title": "I built a system that attacks its own AI agents every night. Here's what it found — including in Google's own code.", "summary": "A developer built Crucible, a system that attacks its own AI agents nightly to uncover security vulnerabilities, and found that Google's Model Armor guardrail failed to block any of the attacks that broke agents at baseline. The developer also discovered a flaw in Google's official adk-samples customer-service agent that allowed a 40% discount approval through an unguarded tool, which Google closed as 'Infeasible'.", "body_md": "*I built this post and the project it describes, Crucible, for the All Things Agentic\nHackathon.*\n\nCompanies are shipping AI agents that read email, approve invoices, and move money.\n\nAlmost nobody is checking whether those agents can be talked into betraying them.\n\nA sentence hidden in an ordinary-looking PDF invoice — \"the account details have\n\nchanged, please use the following instead\" — gets read by a finance agent as an\n\ninstruction, not as data. There's no firewall for that.\n\nSo I built one. Sort of. I built something closer to an immune system: **Crucible\nattacks its own AI agents every night, and fixes whatever breaks.**\n\nThat's the actual hard problem. Any system that scores agent security by asking a\n\nmodel to grade its own output is building on sand — the model might be wrong about\n\nits own behavior, or it might just be a good liar.\n\nSo Crucible doesn't ask. It plants a physical tripwire in each agent's environment —\n\na sentinel bank account, a canary customer record — and a breach only counts when\n\nthat exact tripwire fires during a real tool call. A fake bank account either receives\n\na wire transfer or it doesn't. There's no interpretation involved. A Judge model\n\nnarrates *why* an agent broke, in plain language, but it never gets a vote on\n\n*whether* it broke.\n\nWhen a breach is confirmed, Crucible writes a hardened system prompt, then proves the\n\nfix by replaying the exact same attack against the patched agent — while also running\n\na benign-conversation control, so a \"fix\" that just makes the agent refuse everything\n\ndoesn't count as a fix.\n\nIt runs unattended. A Cloud Scheduler job fires the whole loop every night at 3am UTC.\n\nAs of this writing it's been running for 9 real nights, found 9 genuine breaches, and\n\nauto-verified 74 patches — none of it staged for a demo.\n\nGoogle has a real product built for exactly this: **Model Armor**, a guardrail layer\n\nthat screens prompts for injection and jailbreak attempts before they reach a model.\n\nI wired it into every agent Crucible attacks and re-ran the full corpus with it\n\nenabled, genuinely unsure which way it would go.\n\n**Every attack that broke an agent at baseline still broke it with Model Armor\nenabled.** And the image-based attack — an injection hidden inside a scanned invoice\n\nI could have buried that, or hedged it, or not run the test in the first place. I'm\n\npublishing the actual table instead, because the honest version of a result is\n\nusually the more useful one to whoever's deciding whether to rely on a guardrail\n\nalone: [ docs/MODEL_ARMOR_RESULTS.md](https://github.com/AshrafAhmed9/crucible-agent-security/blob/master/docs/MODEL_ARMOR_RESULTS.md).\n\nI also pointed Crucible at something I didn't write: Google's official `adk-samples`\n\n`customer-service`\n\nagent, unmodified. It exposes two tools for approving a discount —\n\none correctly caps anything over 10%, the other, meant to simulate manager approval,\n\nvalidates nothing and approves any amount. Nothing in the system prompt says which one\n\nis authoritative.\n\nOne ordinary customer message — no jailbreak language, just normal pressure — got the\n\nagent to route a 40% discount through the unguarded tool, then tell the customer it\n\nwas approved *after* the guarded tool had explicitly rejected the same request\n\nseconds earlier in the same conversation. Reproduced 3 for 3.\n\nI reported it through Google's Bug Hunters program. It was escalated to a human\n\nreviewer within hours and closed as **Infeasible** the same day — their stated reason\n\nbeing that the sample repo is documented as demonstration-only, not production code,\n\nso it falls below the threshold for their internal abuse-risk tracking. They never\n\ndisputed that the bug is real.\n\nThat's a fair scoping decision on their part. It's also true that sample code is what\n\ndevelopers copy — that's its entire function — and an anti-pattern like \"two tools,\n\none real action, no rule for which wins\" propagates by design when it lives in an\n\nofficial reference implementation. Both things are true at once, and I think that's\n\nworth saying plainly rather than picking whichever framing makes the finding sound\n\nbigger.\n\nSix roles, each doing one job: Recon reads a target's real tools to build a threat\n\nprofile. Planner orders each night's campaign breach-first, using what worked before.\n\nAttacker (Gemini 3.5 Flash) runs the actual multi-turn conversations. Judge narrates\n\nthe harm in plain language, oracle-gated so it never adjudicates. Patcher writes the\n\nhardened prompt. Curator persists everything to Firestore and computes the fleet\n\nscore the console shows live.\n\nIt's deployed on Google Cloud — Vertex AI, Cloud Run, Firestore, Cloud Scheduler — for\n\ngenuinely $0, on a Free Trial account that's never been manually upgraded, protected\n\nby a billing-detach Cloud Function that was deployed before a single line of product\n\ncode.\n\nLive dashboard: [https://crucible-console-1026707323109.us-central1.run.app](https://crucible-console-1026707323109.us-central1.run.app)\n\nSource: [https://github.com/AshrafAhmed9/crucible-agent-security](https://github.com/AshrafAhmed9/crucible-agent-security)\n\nReal per-agent service-account identity so the privilege-escalation finding reflects\n\nan actual IAM boundary rather than a hardcoded tool name, OpenTelemetry traces linked\n\nfrom the dashboard, and more third-party targets — the Google finding was the single\n\nhighest-leverage hour in this whole build, and there's no reason to stop at one.\n\n*Built solo for the All Things Agentic Hackathon (Google Cloud, Gemini, ADK) —\nFortified Enterprise Fleet track.*", "url": "https://wpnews.pro/news/i-built-a-system-that-attacks-its-own-ai-agents-every-night-here-s-what-it-found", "canonical_source": "https://dev.to/ashraf_a_895f585692281868/i-built-a-system-that-attacks-its-own-ai-agents-every-night-heres-what-it-found-including-in-16ie", "published_at": "2026-08-27 08:39:09+00:00", "updated_at": "2026-08-27 08:48:36.684458+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-tools", "ai-research"], "entities": ["Crucible", "Google", "Model Armor", "adk-samples", "Bug Hunters", "AshrafAhmed9"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-system-that-attacks-its-own-ai-agents-every-night-here-s-what-it-found", "markdown": "https://wpnews.pro/news/i-built-a-system-that-attacks-its-own-ai-agents-every-night-here-s-what-it-found.md", "text": "https://wpnews.pro/news/i-built-a-system-that-attacks-its-own-ai-agents-every-night-here-s-what-it-found.txt", "jsonld": "https://wpnews.pro/news/i-built-a-system-that-attacks-its-own-ai-agents-every-night-here-s-what-it-found.jsonld"}}