AI Harness Engineering · Essay Six · derek wang (derekwang85)
The hardest constraint in this whole pyramid isn't strategy, architecture, contracts, or gates. It's the bottom layer, the one nobody admires: what the system does the second the AI genuinely fails. Every commercial aircraft flies two engines not because it needs both to stay airborne, but because aviation law demands it keep climbing, cruising, and landing on one [ORIGINAL DATA]. Engine redundancy and circuit breakers share one idea borrowed from power engineering: when a circuit overloads, you don't black out the building — you trip that one branch and keep the rest alive.
AI-coding projects need this more than legacy ones, for a brutal reason: AI dies more cleanly than a human does. A tired human slows down, asks for help, leaves a half-finished sentence. An AI stops dead, silently, or pretends it's still fine. The five upper layers govern what the AI produces. This essay is about the layer that governs what survives when the AI produces nothing: resilient implementation. It reduces to four verbs — make it repeat-safe, make it degrade, make it visible, make it recover fast.
For an AI, retrying isn't the exception, it's the default. The network hiccups, the agent can't tell whether its last write landed, so it fires the same request a second time. In a human system a duplicate is a stray log line; in an AI system a duplicate is a second order, a second notification, a second charge. Idempotency is the guarantee that running the same operation twice has exactly the effect of running it once. The test is one question: send the same request twice; did the system do the work twice? If yes, you don't have idempotency.
The cheapest fix I ship over and over is database-native. My trading system's UAT seeded test data with hardcoded IDs — orders/1, kyc/1, quotes/1 — so every reseed broke every reference and the script died on round one [ORIGINAL DATA]. Switching all seed scripts to an upsert gave me repeatability in exchange for one clause:
INSERT INTO orders (id, amount, status)
VALUES (?, ?, 'pending')
ON DUPLICATE KEY UPDATE status = status; -- no-op if the row already exists
Now the agent can retry endlessly; the database always ends up with the same data. When the caller can't tell whether its own write succeeded, hand the system an idempotencyKey and return the first result on every repeated key.
Idempotency handles repeating; degradation handles partial failure. Degradation means the core stays alive at reduced fidelity instead of the whole thing stopping. The user wants "the system is still up"; price precision is a distant second.
My trading system priced everything off an external exchange-rate feed (call it the NCC feed). FMEA flagged the obvious nightmare: NCC is down, and the quote function dies with it, exposing raw errors to the user [ORIGINAL DATA]. The fix had two parts: keep a cached rate as a fallback, and label it. When the feed is healthy, fetch live and cache a copy in the background; when it dies, serve the stale cache and show "rate data may be delayed." Users keep quoting. System keeps running. Cost: a few seconds of staleness.
The label is not decoration. A silent degradation is worse than an error, because downstream systems make decisions on garbage and nobody knows. Degradation is only real if the user sees a warning or the monitor gets an alert. Invisible fallback is no fallback.
Idempotency and degradation you design; visibility you buy with an incident. One day every worker in my multi-agent swarm died at once. Root cause took hours: each worker's startup script used set -e — exit on any error — and combined with flock lock contention, one worker's exit poisoned the lock so every successor failed at boot [ORIGINAL DATA]. Simultaneous death, and because they all failed quietly, no process screamed.
Two lessons. Technically: a daemon's job is to stay alive, not to be strict. set -e belongs to front-of-pipeline scripts where "any failure kills the run" is the goal; a background worker needs explicit error handling — set +e plus its own recovery path. Cognitively: a silent failure is more expensive than a loud one, because a loud one triggers an alert, a retry, a human. Silence triggers nothing, and the system looks healthy until a user discovers the business has been broken for hours. I added a watchdog that polls process health every five minutes and respawns the dead. Manual restart ran thirty minutes armed; the watchdog recovered everything in under five [ORIGINAL DATA].
This is the first principle of the implementation layer: the system may be killed, but it must not be silenced.
The last verb is about the gap between failure and health. A recoverable failure that takes an hour to come back might as well be fatal.
Real scar: MySQL's Docker container died, the backend fell over, and 14 source files surfaced compiler errors one by one like dominoes — a stale import where a controller was deleted but the service wasn't, a renamed DTO field still referenced, a chain of @ConditionalOnProperty beans whose conditions depended on each other [ORIGINAL DATA]. No single error was hard. But fourteen of them queued, and hand-walking the pile cost over 45 minutes of recovery [ORIGINAL DATA]. The value wasn't in fixing fewer errors; it was in the interval. A pre-compile-check.sh gate now runs the whole surface up front and reports the three known families — stale imports, conditional-bean loops, boot smoke test — so the human fixes known types instead of discovering them one 45-minute lap at a time. The same recovery dropped from 45 minutes to under 5 [ORIGINAL DATA].
The bottleneck in recovery is almost never fixing; it's finding. The fastest path back is an automated checker that says "it's one of these three families," then a human who does narrow repairs. When the AI is down, your recovery speed is a function of how fast you can locate, not how well you can patch.
The honest counterpoint: all four verbs can be over-applied. A fallback cache for a feature nothing depends on, a watchdog for a single disposable job, an upsert on a log stream that structurally cannot duplicate — ceremony, not resilience. Every safeguard is itself code that can break, and every watchdog needs watching.
The filter is the same one that runs through this series: tie every resilience measure to a real failure family. The stale-price cache exists because a feed once died. The watchdog exists because a flock once died together. The pre-flight check exists because a cascade once cost the team an afternoon. So the discipline is: if you can't name the past incident that justifies a resilience feature, you don't get to add it yet. You'll know the real one when it fires. The four verbs are cheap only when they answer a debt you actually owed — and the moment that's true, the business keeps moving precisely because the system was built to be killed, not silenced.