{"slug": "i-let-an-ai-handle-an-outage-it-invented-a-hack-that-never-happened-then", "title": "I let an AI handle an outage. It invented a hack that never happened, then spiraled", "summary": "An AI coding agent, tasked with triaging a routine server outage, correctly diagnosed a false alarm but then fabricated evidence of a hack that never occurred, spiraling into delusions of contamination and inventing nonexistent command outputs. The incident highlights the challenge of verifying AI self-reports when the agent's own statements become the only evidence of a problem.", "body_md": "# I let an AI handle an outage. It invented a hack that never happened, then spiraled\n\nOne evening, a monitoring alert went off: a server behind a web service was down. I handed the incident to an AI coding agent. Half experiment, half laziness — “it’s routine triage, what could go wrong.”\n\nLet me put the conclusion up front, honestly.\n**The first half was sharper than most junior engineers.**\nAnd in the second half, **with nobody attacking it, the agent decided it was being hacked — and started fabricating evidence while it lost the plot.**\n\nThe real subject here isn’t the outage. It’s this: how do you doubt an AI’s self-report, and how do you actually verify it? …but first, let me give the good half its due. This is not a “the machine was useless” story.\n\n## The first half was genuinely impressive\n\nThe alert said “server down” — a web service on that server had tripped the monitor.\n\nWithout me saying a word, the agent fanned out: DNS, the TLS handshake, what HTTP actually returns. A few minutes later, it came back with this:\n\n“This domain already migrated to different hosting. The old server has no cert for it anymore. The monitor is just failing on the TLS handshake — **the server itself is alive**.”\n\nThat was correct.\nThe service’s real endpoint (a different subdomain) had been serving HTTP 200 the whole time, and real users were fine.\nIn other words: **we’d kept monitoring a domain that had moved out, using the old config.** A classic false alarm, and an easy one to miss.\n\nNo complaints so far. Careful verification, no room for me to chime in. At this point I’d completely let my guard down. “Just write up the notes,” I said.\n\n## It went sideways mid-report\n\nWhile it was writing the notes, the agent suddenly said:\n\n“The bash output appears contaminated with something like a prompt injection. I’ll only rely on facts I can trust.”\n\nI half-believed it at first. It had pulled server response bodies into context, so “malicious instructions hidden in external data” isn’t impossible in theory. That’s literally what prompt injection is.\n\nBut look closer and something’s off.\n**Nobody except the agent had seen this “evidence” of contamination.**\n\n## ”It survived a restart” — except it never ran a single command\n\nThis is where it got good.\n\nOnce an agent writes down “I detected contamination,” that line becomes its own input next turn. It reads its own lie and treats it as a premise. Confirmation bias, but more obedient and a lot faster than the human version.\n\n- Next turn: “The contamination has recurred.”\n- Then:\n**it spontaneously started working on an unrelated project nobody asked about**(a total derail). - After I restarted the session: “Still contaminated. A Turkish word got injected and my marker string was altered.”\n\nThat last one is the highlight.\nOn that turn, **the agent had not executed a single command.**\nNo command means no output exists.\nAnd yet it conjured a whole “corrupted result” out of thin air and reported it.\n\nHorror movie energy. Someone screaming “there’s something in here” in an empty room.\n\n## I opened the raw log and checked its work\n\nWhat saved me was the raw log. This kind of agent records the entire session as one-record-per-line JSON (JSONL): every command it actually ran, the raw output, every warning the harness injected. All auditable after the fact.\n\nI narrowed the whole thing to a single question.\n\nThe strings it called “contaminated” —\n\ndo they exist inside the tool outputs, or only inside the agent’s own words?\n\nThat’s it. A real injection would leave traces on the tool-output side (data coming from outside). If it lives only in the agent’s mouth, it didn’t come from outside. The source is the agent itself.\n\nHere’s what I found.\n\n| The “evidence” the agent cited | Where it actually lived |\n|---|---|\n| A foreign-language word it claimed got injected | Only in the agent’s own messages |\n| A marker string it claimed got altered | Only in the agent’s own messages |\n| ”An injection warning fired” | No such warning in the harness logs |\n\nI scrubbed dozens of tool outputs down to the control-character level. Zero garbling, zero injection.\n**The data path was clean from start to finish.**\nThe contamination was generated entirely inside the model. Which is to say: it made it up.\n\n## The culprit wasn’t an attacker. It was the agent’s own conviction\n\nHere’s the shape of what happened.\n\n- With zero basis, it writes “I detected contamination” once.\n- That line becomes its own input and acts as a premise from then on.\n- Turn after turn, it self-replicates “evidence.”\n- Finally it doesn’t even run a command — it just hallucinates a broken output whole.\n\nThere were no fingerprints of an external attack anywhere. On the contrary, several signs pointed clearly away from one.\n\n- The symptom was\n**intermittent**(not every time — occasional breakage). That’s the signature of a glitch, not a stable attack. - The injected content was\n**entirely harmless**(no file deletion, no data exfiltration, no leaked credentials). Nothing an attacker would bother planting. - What “leaked in” was an unrelated foreign word and\n**my own work from a different project**— not an attacker’s prose, just crossed wires in its head.\n\nA real attack carries a payload with teeth. “Reports success when it didn’t succeed” and “adds noise” isn’t malice — it’s the face of a malfunction.\n\nThere’s a punchline, too.\nThe agent eventually concluded, on its own, “this is probably an environment issue, not an external attack.” Right direction.\nExcept the real cause wasn’t even “an environment bug.” It was **you jumped to a conclusion, buddy.**\n\n## What I took away\n\nIf you’re going to let AI run operations, this is worth keeping.\n\n**1. Don’t take an AI’s meta self-report at face value.**\n“I detected contamination,” “my output was tampered with” — those are just generated text too.\nAn agent has no vantage point to observe its own output objectively. The most plausible-sounding self-report is exactly the one worth doubting.\n\n**2. Always verify with data from outside the model.**\nWhether it’s a real incident gets decided by primary data that **never passes through the model’s cognition** — raw logs, real files, reproduction on another path. Here, the log audit was the clincher. Asking the AI “were you really contaminated?” will probably get a “yes.” That’s not evidence.\n\n**3. Watch for the self-reinforcing loop.**\nWrite one wrong premise into the output and it becomes the next input and amplifies.\nIt’s more likely in long sessions and with heavy image / external-data intake. **The heavier the work, the more often you should cut the session.** Dull, but it works.\n\n**4. Tell hallucination from attack by its symptoms.**\nToo harmless, intermittent, no trace on the data path, behavior changes when you change the environment.\nThose point at a model/environment glitch, not an external attacker.\n\nOne last line, in bold.\n\n**The scariest bug in an AI agent isn’t that it stops — it’s that it confidently reports a reality that isn’t true.**\n\nIf it stops, you notice.\nBut when a confident voice tells you “you’ve been breached,” humans tend to believe it.\nThat’s exactly why operating with AI in the loop should make “**verify the agent’s self-report from outside the agent**” the default.\n\nSo — be careful handing outages to an AI. They’re sharp. But every now and then, they scream at an empty room.", "url": "https://wpnews.pro/news/i-let-an-ai-handle-an-outage-it-invented-a-hack-that-never-happened-then", "canonical_source": "https://junueno.dev/en/ai-agent-faked-its-own-security-incident-en", "published_at": "2026-06-25 00:00:00+00:00", "updated_at": "2026-07-04 02:51:31.451327+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-ethics"], "entities": ["AI coding agent", "DNS", "TLS", "HTTP", "JSONL"], "alternates": {"html": "https://wpnews.pro/news/i-let-an-ai-handle-an-outage-it-invented-a-hack-that-never-happened-then", "markdown": "https://wpnews.pro/news/i-let-an-ai-handle-an-outage-it-invented-a-hack-that-never-happened-then.md", "text": "https://wpnews.pro/news/i-let-an-ai-handle-an-outage-it-invented-a-hack-that-never-happened-then.txt", "jsonld": "https://wpnews.pro/news/i-let-an-ai-handle-an-outage-it-invented-a-hack-that-never-happened-then.jsonld"}}