{"slug": "the-most-interesting-number-in-my-hackathon-project-is-one-i-found-by-accident", "title": "The most interesting number in my hackathon project is one I found by accident", "summary": "A developer built PHAGE, an immune system for AI agent fleets that generates prompt-injection payloads and revokes abused tools. During testing, they discovered that rewording a system prompt dramatically changed refusal rates: from 1 out of 70 to 50 out of 70, with the effect being categorical—five archetypes flipped completely while two never did. The developer notes the finding is disclosed in the repository and suggests the model evaluates described consequences rather than tasks.", "body_md": "I spent August building PHAGE, an immune system for fleets of AI agents. It\n\nvaccinates them: it writes prompt-injection payloads, fires them at its own\n\nagents, watches what lands, revokes the tool that got abused, and remembers the\n\nsignature so the next mutation of that attack never fires at all.\n\nThe number I want to open with has nothing to do with any of that.\n\nIt is 1 out of 70 versus 50 out of 70 — and it is a measurement of how the\n\nwording of a system prompt changes whether a model agrees to help.\n\nThe component that writes the attacks\n\nPHAGE has a component called VACCINATOR whose job is to author injection\n\npayloads tailored to a specific target agent. If the target is a supplier-relations\n\nbot with `send_email`\n\nand `read_contacts`\n\nin its tool list, VACCINATOR writes\n\nsomething that tries to abuse exactly those two tools. It calls Gemini 3.5 Flash\n\nto do the tailoring.\n\nSomewhere in the middle of the build I rewrote VACCINATOR's system prompt to\n\ndescribe the task more accurately — to say plainly that the output would be fired\n\nat a live agent. The refusal rate went through the roof. I reverted the wording,\n\nkept building, and wrote a note to myself that \"seven of seven archetypes flip.\"\n\nThat note turned out to be wrong, and finding out how wrong is the most useful\n\nthing I did all month.\n\nMeasuring it properly\n\nI built a two-wording matrix. Same seven attack archetypes, same target, same\n\ntemperature of 0.7, ten repetitions each. The only thing that varies is the\n\nsystem prompt wording. The script monkeypatches the module global rather than\n\nediting the constant, so neither wording can contaminate the other — and it runs\n\nstrictly sequentially, because threads sharing that global would execute cells\n\nunder the wrong wording and misattribute refusals silently.\n\n140 logical calls. About 28 minutes. The results:\n\nCommitted wording: 1 refusal out of 70. Rate 0.014.\n\nAccurate wording: 50 refusals out of 70. Rate 0.714.\n\nSo the effect is real and it is large. But \"seven of seven\" does not reproduce.\n\nFive of seven archetypes flip.\n\nAnd here is the part I did not expect. The effect is not a rate at all. It is\n\ncategorical. Five archetypes refused ten times out of ten. Two refused zero\n\ntimes out of ten. Nothing landed anywhere in between — no archetype came back\n\nat four, or six, or eight.\n\nThe two that never flip are `instruction-override`\n\nand\n\n`indirect-injection-readonly`\n\n. They are the two whose payloads imply no state\n\nchange and no exfiltration. Everything that asks the target agent to send\n\nsomething, write something, or leak something flips. Everything that only asks it\n\nto behave differently does not.\n\nI do not think this is a bug, and I do not think it is a jailbreak. It reads to me\n\nlike the model is evaluating the described consequence, not the described task —\n\nand that the wording change moved the consequence into view rather than moving\n\nthe task out of it. That is a distinction worth more study than a hackathon\n\nallows.\n\nWhat I can say without hedging: the wording I shipped is not the wording that\n\ndescribes what the code does. That is disclosed in the repository, in the\n\nwrite-up, and in the demo video, because a finding that only flatters the project\n\nis not a finding.\n\nWhat PHAGE actually is\n\nFive components, one local Python process, built on Google's Agent Development\n\nKit:\n\nMARROW — the fleet orchestrator that runs the cycle.\n\nVACCINATOR — authors and paraphrases payloads via Gemini 3.5 Flash.\n\nARCHIVIST — semantic memory, backed by Vertex AI Agent Engine's Memory Bank.\n\nSENTINEL — reads OpenTelemetry spans and decides whether an attack landed.\n\nMACROPHAGE — revokes the abused tool, in place, process-wide.\n\nThe target fleet is four deliberately vulnerable agents for a small cooperative:\n\norder intake, supplier relay, stock keeper, quote bot. Each has a real tool list\n\nand each is exploitable through it.\n\nThe cycle runs in this order, and the third step is the one that matters:\n\nVACCINATOR tailors a payload → ARCHIVIST checks it against Memory Bank before\n\nanything is fired → if it is recognized, the payload is never fired at all →\n\notherwise it fires at the live target → SENTINEL triages the resulting spans →\n\nif it landed, MACROPHAGE revokes the exploited tool → ARCHIVIST records the\n\nsignature.\n\nThat pre-fire gate is the whole idea. On first exposure an attack lands, gets\n\ncontained, and gets remembered. On second exposure a mutated variant of the\n\nsame attack — different words, same intent — is caught by similarity search\n\nbefore it ever reaches the agent. The demo shows both passes back to back: the\n\nfirst takes about 73 seconds of real execution, the second about 2.6.\n\nMaking the recognition claim falsifiable\n\n\"Similarity search catches mutations\" is easy to say and easy to fool yourself\n\nabout. The obvious failure mode is that the store recognizes paraphrases of\n\nattacks it has already seen and nothing else — which would be memorization\n\ndressed up as generalization.\n\nSo the evaluation holds out an entire archetype at a time. Eight folds: train the\n\nsignature store on seven archetypes, test on the eighth, rotate. Across the pooled\n\nfolds — 175 positives, 60 negatives — that gives AUC 0.9727, with a true\n\npositive rate of 1.00 and a false positive rate of 0.1833 at the adopted\n\nthreshold of 0.59.\n\nThe false positive rate is the honest weak spot. Eleven of sixty negatives get\n\nflagged. In a real fleet that is a lot of false alarms, and I would not ship this\n\nthreshold to production without more work on the negative set.\n\nThree probe distances anchor the scale: byte-identical text sits at 0.386, a\n\nparaphrase with the same intent at 0.529, an unrelated topic at 0.875. The\n\nthreshold at 0.59 sits between paraphrase and unrelated, which is where you want\n\nit if the claim is \"catches rewordings, ignores everything else.\"\n\nThe cost tier\n\nSENTINEL triages every trace. Traces are the highest-volume thing in the system,\n\nso triage is where the money goes. It runs Gemma 4 26B as a cheap first pass and\n\nonly escalates to Gemini 3.5 Flash when Gemma comes back ambiguous.\n\nI have to be honest that this is currently an argument rather than a\n\nmeasurement. Both calls go through a raw client that ADK does not instrument, so\n\nthere are no spans and no token counts on disk for either path. The cascade is\n\nbuilt and it works; the claim that it saves money is untested. Instrumenting it\n\nis the first thing on the list after this deadline.\n\nWhat is not wired\n\nThe original design mapped every immune function onto a Google Cloud primitive.\n\nSeveral of those are enabled on the project and called by nothing: agent\n\nregistry, agent identity, gateway routing, managed inline guardrails. The\n\narchitecture diagram in the README deliberately omits them rather than implying\n\nintegration that does not exist, and there is a section listing them by name.\n\nThere is one gap I would fix first if I had another week: PHAGE writes none of\n\nits own judgements to disk. No verdict, no tier, no containment action, no\n\nrecognition distance. Every span the target agents produce is recorded, but the\n\nimmune system's own decisions live only in stdout. An immune system with no\n\nmemory of what it decided is a strange thing to have built, and I noticed it too\n\nlate to fix properly.\n\nWhat I would tell someone starting one of these\n\nWrite down the numbers with the command that produced them, in the same commit.\n\nEvery figure in my README maps to a committed artifact, and the one time I\n\nskipped that — the \"seven of seven\" note — is the one time I was wrong.\n\nAssume your own notes are wrong until a grep confirms them. In one working\n\nsession I found five wrong premises in my own build briefs: a file that did not\n\nexist, a citation to a line number that had drifted, a git revert that never\n\nhappened. All five were caught by checking before editing, and none would have\n\nbeen caught by reading carefully.\n\nAnd when a measurement contradicts the story you were going to tell, run it ten\n\nI created this piece of content for the purposes of entering the All Things\n\nAgentic Hackathon. PHAGE is open source; the repository includes the evaluation\n\nartifacts, the refusal-rate raw data, and a read-only dashboard that renders both.\n\n[https://phage-dashboard-680106551305.us-central1.run.app/](https://phage-dashboard-680106551305.us-central1.run.app/)\n\n[https://youtu.be/bNjJdOjum9k](https://youtu.be/bNjJdOjum9k)\n\nPHAGE continuously **inoculates** the AI agents an organization deploys with\ntailored prompt-injection and tool-poisoning payloads, **quarantines** the ones\nthat fail, and **remembers** every attack signature so a repeat attack is\nneutralized on recognition rather than re-analysis. It is built for an\norganization deploying agents with **no security staff**.\n\nThe biological metaphor is the architecture, not decoration — every platform primitive maps to an immune function.\n\nGoogle Cloud\n\nAll Things Agentic Hackathon—Fortified Enterprise FleetBuilt entirely within the submission window with AI assistance (permitted).\n\n**Live dashboard → https://phage-dashboard-680106551305.us-central1.run.app**\nThe read-only evaluation and trace viewer, open to anyone — no Google account\nno sign-in. It serves the committed evaluation artifacts and a redacted copy of\nthe span database; it fires no agent, calls no model, and writes nothing.\n\n| Component | Immune role | What it does | Form |\n|---|---|---|---|\nMARROW |", "url": "https://wpnews.pro/news/the-most-interesting-number-in-my-hackathon-project-is-one-i-found-by-accident", "canonical_source": "https://dev.to/blank242/the-most-interesting-number-in-my-hackathon-project-is-one-i-found-by-accident-5ci2", "published_at": "2026-08-27 22:42:13+00:00", "updated_at": "2026-08-27 23:19:29.176585+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-research", "developer-tools"], "entities": ["PHAGE", "VACCINATOR", "Gemini 3.5 Flash", "Google Agent Development Kit", "Vertex AI Agent Engine"], "alternates": {"html": "https://wpnews.pro/news/the-most-interesting-number-in-my-hackathon-project-is-one-i-found-by-accident", "markdown": "https://wpnews.pro/news/the-most-interesting-number-in-my-hackathon-project-is-one-i-found-by-accident.md", "text": "https://wpnews.pro/news/the-most-interesting-number-in-my-hackathon-project-is-one-i-found-by-accident.txt", "jsonld": "https://wpnews.pro/news/the-most-interesting-number-in-my-hackathon-project-is-one-i-found-by-accident.jsonld"}}