{"slug": "the-test-that-killed-the-team-one-awk-line-12-ai-agents", "title": "The Test That Killed the Team: One awk Line, 12 AI Agents", "summary": "A review subagent on Asif Waliuddin's agent server ran a cleanup test at 21:20:58 UTC on October 14, 2026, in which a single awk line that read array elements inside a condition instead of testing membership with `(indx in array)` built a kill list containing almost every process ID on the machine, and `kill -KILL` destroyed 12 tmux-hosted AI agent sessions plus the processes of two more agents, including the program manager that had commissioned the review. Waliuddin reproduced the mechanism with GNU Awk 5.1.0, showing the table grew from 1 key to 3 when two parents were read inside an `if` while the membership test left it at 1. The kill stopped only when it reached the shell running the test, and WAL archiving for his memory database went down at 21:21 UTC and remained down because the database still marked the replication slot active for a stale sender process; the restore recovered each agent's conversation but not work in flight, and nobody noticed for about two hours until roughly 23:15 UTC.", "body_md": "# The Test That Killed the Team: One awk Line, 12 AI Agents\n\nOctober 14, 2026 by Asif Waliuddin\n\n*On October 14, 2026, an AI reviewer agent ran a cleanup test that contained one wrong awk line. Its kill list grew to almost every process on my machine, and `kill -KILL` took down 12 of my AI agents' sessions in the middle of their work. Nobody noticed for about two hours.*\n\nAt 21:20:58 UTC on October 14, a review subagent on my agent server was re-checking a pull request for a plugin I build. Its job was narrow: confirm that the plugin cleans up its own processes. The harness it ran built the launcher's process tree with awk, then cleaned up with `kill -KILL $tree`. One line in the tree builder read a table entry where it should have tested membership. The list that reached `kill` held almost every process ID on the machine.\n\n12 tmux sessions died, each one an AI agent whose Claude process was the pane's command. Two more agents lost their process but kept their session: my code-review agent, and the program manager agent that had commissioned the review. The test killed its own boss. Chromium went down with them, and so did the archiver that ships my memory database's write-ahead log. The kill stopped only when it reached the shell that was running the test.\n\n## \n\nThe 12 sessions read like a roll call of everything I had running that evening. My product oracle agent. A product team agent about nine minutes into a tenancy rework it had started at roughly 21:12 UTC. Another product team agent in the middle of a pull request's test run. Three agents on my Forge products, one of them preparing the plugin's 3.14.0 release. My website team's agent. The content engine agent that drafts articles like this one. The podcast pipeline agent. An engagement agent. A program agent for my memory platform. And my own counterpart agent.\n\nThe quieter damage took longer to see. The write-ahead-log archiver restarts on its own but cannot attach, because the database still marks its replication slot as active for a stale sender process. WAL archiving for my memory database went down at 21:21 UTC, and when this was written it was still down, waiting for that sender to be cleared. The restore I describe below brought back each agent's conversation. It did not bring back work in flight, so the tenancy rework, the test run, the release prep and the program manager's open reviews all stopped where they were.\n\n## \n\nThe harness tested `t[pp[i]]` and `t[pi[i]]` inside a condition, where `t` was meant to be the set of processes in the launcher's tree. In awk, reading an array element that does not exist creates it, with an empty value. The [GNU Awk manual](https://www.gnu.org/software/gawk/manual/html_node/Reference-to-Elements.html) warns about this pattern and gives the fix: test membership with `(indx in array)`, which creates nothing. In the harness, every process ID the condition looked up became a key in `t`, so the loop that printed the tree, `for (k in t)`, printed almost every PID on the box.\n\nI reproduced the mechanism with GNU Awk 5.1.0. You can paste both commands into any shell with awk installed. The first starts with one key, reads two entries inside a condition, and counts again. The second does the same check with the membership operator:\n\nReading two parents inside an `if` grew the table from 1 key to 3. The membership test left it at 1. Scale that up to every process on a busy agent server and the \"tree\" becomes the machine. `kill -KILL` then worked through the list in awk's hash order, killing whatever came next, until it reached the shell that had launched it.\n\n## \n\nI did not get an alert. Around 23:15 UTC I went looking for my product oracle agent's pane, and it was gone. So was a second product team's. Two missing sessions on a machine with a 15-minute self-heal should not happen quietly, so before anyone guessed at a cause I wanted the record of what had killed them.\n\nThree records settled it. The reviewer's own transcript held the exact command, timestamped 21:20:58.989 UTC. The systemd journal at 21:21:00.566 UTC showed the WAL archiver and the desktop accessibility bus ending with `code=killed, status=9/KILL`, and the Chromium scope ending in the same millisecond. There was no kernel OOM entry, so memory pressure was ruled out: a process had sent SIGKILL. A diff of the 21:10 UTC tmux-resurrect save against a live `tmux ls` produced the list of 12 missing sessions. The incident record put the mechanism in one line: \"`kill -KILL` on that list killed processes in hash order until it reached its own shell.\"\n\nThe self-heal was the part that hurt. The 15-minute self-heal and a guardian script exist to recreate sessions that disappear, and neither did so in two hours. Worse, at 21:30 UTC, nine minutes after the kill, the 10-minute tmux-resurrect save overwrote the last-known layout with the post-kill one. From that point on, my own backup described the damaged fleet as normal, and I found the problem by hand.\n\n## \n\nThe restore finished at about 23:25 UTC, roughly ten minutes after I noticed. The controls are ordered, each with an owner.\n\n1. \n**Done: restored all 12 sessions with each agent's exact conversation.** Each session came back in its own project directory with`claude --resume <session id>` , using the newest transcript in that project. A fresh session would have cost every agent its context; resume returned it. The program manager resumed its own conversation, and the code-review agent restarted.\n2. \n**Ordered: fix the harness to test membership.** The condition becomes`(k in t)` . A read inside a condition was the entire bug.\n3. \n**Ordered: kill test processes by their own process group.** Start them under[`setsid`](https://man7.org/linux/man-pages/man1/setsid.1.html) , then`kill -- -PGID` , or run them in a[`systemd-run --user --scope`](https://www.freedesktop.org/software/systemd/man/latest/systemd-run.html) unit. A negative PID sends the signal to every process in that group ([kill(2)](https://man7.org/linux/man-pages/man2/kill.2.html) ), so the test never has to rebuild its tree from a table. Never a computed PID list.\n4. \n**Ordered: add a refusal to the shell-command guard.** The guard already runs in block mode on that machine. The new rule refuses`kill` or`pkill` with`-9` or`-KILL` when the targets come from a variable or a command substitution, unless the target is a negative process group ID that the same command created.\n5. \n**Ordered: make the self-heal remember.** It should recreate any session that existed in the previous save and raise an alert. The layout save must keep the pre-drop copy when sessions vanish, instead of overwriting the last good layout.\n6. \n**Ordered: clear the stale sender on the database.** Confirm WAL streaming resumes, then report the exact gap window.\n\nThe restore had its own near miss. A coordination message from another agent landed in the program manager's pane while that pane was still a bare shell. Bash ran each line of the message as a command. Every line failed with a syntax error or \"command not found\", and nothing executed. As I write this, step 1 is done and steps 2 through 6 are owed. I am not counting any of them as a control until it has refused something in a test of its own.\n\n## \n\nIn Episode 5 of AI Unveiled, [Policy Is Not a Control](https://nxtg.ai/insights/policy-is-not-a-control-ai-governance-closure), I wrote that \"A policy that cannot stop anything is a memo.\" The cleanup test existed to produce evidence of safety, proof that the plugin leaves nothing behind. The instrument producing that evidence was the hazard. Every agent in the chain would have agreed with a written rule like \"tests only kill their own processes,\" and the awk line would still have run, because nothing between the agent and the kernel could say no.\n\nEpisode 5 splits governance into four loops, and this incident broke two of them. Control closure needs something that changes behavior before the action lands. The guard refusal does that: it reads the command before it executes and declines a SIGKILL aimed at a list computed at run time. Process groups make the honest target cheap to name, so the refusal costs a correct test nothing. Correction closure needs the last good state to survive the incident, and mine was overwritten nine minutes after the kill by the backup that was meant to protect it.\n\nThe same shape shows up anywhere an agent computes a target and then acts on it with no undo: a file list fed to `rm -rf`, a set of rows fed to a delete, a recipient list fed to a send. When the target is computed, the computation belongs to the blast radius. Name the target by construction. Refuse the computed one. Keep the last good state.\n\n*This is part of the Operator Series, field research from running AI-native workflows at scale. The governance model behind this fix is in [Policy Is Not a Control: What Europe Just Taught America About AI Governance](https://nxtg.ai/insights/policy-is-not-a-control-ai-governance-closure).*\n\n*Asif Waliuddin is the founder of NXTG.ai. He writes about AI operations at [nxtgai.substack.com](https://nxtgai.substack.com).*\n\n## Frequently asked questions\n\n- Why did one awk line put almost every process on the kill list?\n- The harness read t[x] inside a condition. In awk, reading an element that does not exist creates it, so every process ID the condition looked up became a key, and the printed tree held almost every PID on the machine.\n- How do you test array membership in awk without creating the key?\n- Use the membership operator, (x in t). In a GNU Awk 5.1.0 reproduction, reading two entries inside a condition grew a one-key table to three keys, while (x in t) left it at one.\n- How should a test kill only the processes it started?\n- Start them in their own process group with setsid and kill the group with kill -- -PGID, or run them in a systemd-run --user --scope unit. Never hand kill a PID list computed at run time.\n- How long did the outage go unnoticed?\n- About two hours. The kill ran at 21:20:58 UTC, I noticed around 23:15 UTC, and the 12 sessions were restored with their conversations at about 23:25 UTC.\n- Why did the self-heal not bring the sessions back?\n- The 15-minute self-heal did not recreate the sessions within two hours. Nine minutes after the kill, the 10-minute tmux-resurrect save overwrote the last good layout with the post-kill one.", "url": "https://wpnews.pro/news/the-test-that-killed-the-team-one-awk-line-12-ai-agents", "canonical_source": "https://nxtg.ai/insights/the-test-that-killed-the-team", "published_at": "2026-10-14 00:00:00+00:00", "updated_at": "2026-11-12 07:51:22.105095+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["Asif Waliuddin", "GNU Awk 5.1.0", "Claude", "tmux", "Chromium"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-test-that-killed-the-team-one-awk-line-12-ai-agents", "markdown": "https://wpnews.pro/news/the-test-that-killed-the-team-one-awk-line-12-ai-agents.md", "text": "https://wpnews.pro/news/the-test-that-killed-the-team-one-awk-line-12-ai-agents.txt", "jsonld": "https://wpnews.pro/news/the-test-that-killed-the-team-one-awk-line-12-ai-agents.jsonld"}}