Opinion: AI Server Changes Need a Fault Drill, Not Just a Rollback Plan

A developer argues that AI-generated server patches should be validated with fault drills on disposable servers before human review, rather than relying solely on rollback plans. The workflow uses free model access to generate failure hypotheses and a script to test them, approving changes only when all tested failure modes are recoverable and data remains intact.

A rollback plan tells you how to undo an AI change, but not what breaks first when the change stays in place. Most production incidents do not begin with a deliberate rollback; they begin with an unexpected failure mode that the author never tested. I now treat a passing fault drill as a precondition for reviewing any AI-generated server patch. The drill runs on a disposable server before a human reads a single line of the diff. A rollback plan answers a question about the past: how do we return the system to a known state? A fault drill answers a question about the future: what happens when this change meets a condition the author did not imagine? The second question decides whether you get paged at 3 a.m. A change with a perfect rollback can still fail in a way that nobody notices until the data is gone. Free model access changes the economics of this argument, because generation stops being the bottleneck and verification starts. When a draft is nearly free, the cheapest verification is the one that breaks the change on purpose. A rollback plan is documentation; a fault drill is evidence. Documentation tells you what should happen, while evidence tells you what actually happens on a real service manager. The workflow assumes two cheap resources: a model that generates failure hypotheses from a diff, and a server that can be destroyed after the drill. MonkeyCode's free model access covers the first, and its free server option covers the second, so a drill costs almost nothing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Any ephemeral VM or container host works if you prefer a different provider. Ask the model to enumerate failure modes for the diff, and forbid it from proposing fixes, because fixes are a distraction at this stage. The prompt below is the one I use, and it produces a catalog that the drill can test. Treat the model's list as a hypothesis set, not a safety analysis. The list is useful because it is cheap and broad, and it is dangerous only if you trust it without running the drill. You are reviewing a systemd unit change. Here is the diff: <diff List the five most likely failure modes of this change on a running server. For each failure mode, state: 1 the trigger, 2 the observable symptom, 3 whether a plain restart recovers the service, 4 whether data can be lost. Do not propose fixes. Provision a server that mirrors the target's OS and service manager, apply the change, and confirm that the service starts cleanly. The server must be disposable, because the next step will deliberately break it and you should not care. A shared staging box is the wrong target, since a drill that corrupts another team's state teaches the wrong lesson. Run fault-drill.sh against the unit and a sentinel file that represents data the service must not lose. The script applies each fault, waits a few seconds, records the service state, attempts a manual restart, and checks the sentinel. Each fault is applied to a restored system, so one bad result does not contaminate the next row. Each output row classifies one failure mode along three axes: service state after the fault, recovery after a manual restart, and sentinel integrity. A row with down , manual-fail , and lost is a reject, while a row with active , manual-ok , and intact is boring. Boring is the highest compliment a failure mode can earn, because boring means the incident is already solved by existing machinery. Approve the change only when every tested failure mode is boring or recoverable, and document the verdict next to the diff. If any fault loses the sentinel, the change fails review regardless of how clean the diff looks. A clean diff is a statement about intent, while a drill result is a statement about behavior, and behavior is what runs in production. The script is intentionally small, because a drill you cannot read in one sitting is a drill you will not run. It takes a unit name, a sentinel path, and a list of faults, and it prints a classification table. Run it only on the disposable server, and restore the unit between faults so the results stay independent. Treat the script as a starting point, not a certification suite, and adjust the fault list and the fill size to your unit and disk before you trust a verdict. bash /usr/bin/env bash fault-drill.sh — classify failure modes of a systemd unit on a disposable server Usage: ./fault-drill.sh <unit <sentinel fault ... set -euo pipefail UNIT="$1" SENTINEL="$2" shift 2 CONFIG="$ systemctl show -p FragmentPath --value "$UNIT" " CONFIG BAK="${CONFIG}.drill-bak" restore { -f "$CONFIG BAK" && mv "$CONFIG BAK" "$CONFIG" rm -f /tmp/drill-fill systemctl daemon-reload 2 /dev/null || true systemctl reset-failed "$UNIT" 2 /dev/null || true } printf '%-14s %-8s %-12s %s\n' "FAULT" "STATE" "RECOVERY" "SENTINEL" for fault in "$@"; do case "$fault" in kill systemctl kill --signal=SIGKILL "$UNIT" || true ;; stop systemctl stop "$UNIT" || true ;; restart systemctl restart "$UNIT" || true ;; corrupt cp "$CONFIG" "$CONFIG BAK"; printf 'garbage\n' "$CONFIG" ;; fill-disk dd if=/dev/zero of=/tmp/drill-fill bs=1M count=256 status=none || true ;; revoke cp "$CONFIG" "$CONFIG BAK"; chmod 000 "$CONFIG" ;; echo "unknown fault: $fault" &2; exit 2 ;; esac sleep 3 if systemctl is-active --quiet "$UNIT"; then state="active"; else state="down"; fi if systemctl start "$UNIT" 2 /dev/null; then recovery="manual-ok"; else recovery="manual-fail"; fi if -s "$SENTINEL" ; then sentinel="intact"; else sentinel="lost"; fi printf '%-14s %-8s %-12s %s\n' "$fault" "$state" "$recovery" "$sentinel" restore done Example invocation against a unit that has already been changed on the disposable server: ./fault-drill.sh myapp.service /var/lib/myapp/data.db kill stop restart corrupt fill-disk revoke Sample output for a change that survives the boring faults but loses data when the disk fills: FAULT STATE RECOVERY SENTINEL kill active manual-ok intact stop down manual-ok intact restart active manual-ok intact corrupt down manual-fail intact fill-disk down manual-ok lost revoke active manual-fail intact Note what the table does that a diff review cannot: it separates the failure modes a restart absorbs from the ones that destroy data. The fill-disk row is the reason this change fails review even though every other fault was boring. The revoke row is the subtle one, because the service still reports active while a restart fails, so the next reboot turns a healthy unit into a permanently down one. Apply the verdict of the worst row, not the average, because one lost row dominates any number of intact rows. Data loss is the only outcome that a rollback plan cannot undo, which makes it the natural veto for the whole drill. | Worst row observed | Verdict | |---|---| | active + manual-ok + intact | boring; approve and move on | | down + manual-ok + intact | recoverable; add an alert for the trigger | | active/down + manual-fail + intact | fragile; require a guard or a fix before approval | | any row with lost | reject the change | The drill is a heuristic, not a chaos-engineering platform, and it tests a handful of faults rather than the full failure space. It does not exercise load, latency, multi-node coordination, or the behavior of a degraded dependency. The free model's failure-mode list is a hypothesis generator, and every hypothesis must be confirmed by the drill before it counts as evidence. Skip this workflow if you have no disposable environment, because drilling production or shared staging is worse than not drilling at all. Skip it if your change touches multiple hosts, because a single-node drill will give you false confidence. Skip it if your team cannot act on a lost row, because a drill without a rejection rule is just theater. A rollback plan is the floor, not the ceiling, for reviewing AI-generated server changes. The ceiling is a fault drill that shows you which failure modes are boring and which ones lose data. Free model access makes the hypothesis list cheap, and a free disposable server makes the evidence cheap. The only expensive part is the discipline to reject a change that fails one row. If your team runs a drill like this, publish the boring failure modes you catalog; they are worth more than the impressive ones.