My agent dry-ran fine in staging 100 times — then wrecked production on the first real run

A developer building AI agents on Cloudflare Workers discovered that staging-to-production data bleeds caused production outages despite 100 successful dry runs. The engineer found that environment drift and non-deterministic agent execution paths led to fresh tool call sequences in production, while mock responses in dry-run mode created orphaned records when writes to D1 were not scoped. The fix involved propagating a dry-run flag via KV across all subsequent writes in the same run, and the developer warns that hook failures can bypass dry-run protection entirely.

A staging-to-production data bleed cost me 4 hours of rollback. That's what finally made dry-run a structural requirement, not an afterthought. The common advice is: test in staging, promote when green. The problem is environment drift. My D1 schema changes once or twice a week, and a solo operator can't keep staging perfectly synchronized. Worse, agents don't have fixed execution paths — the same input can produce a different tool call sequence on the next run. I ran a flow 100 times in staging and still hit a fresh path on the first production execution. The most surprising thing I learned after 6 months of running this: latency wasn't the problem I expected . KV writes averaged 12ms — basically imperceptible. The real problem was that mock responses fool the agent into treating skipped writes as real successes. I'd dry-run an R2 put , the agent would believe the file was uploaded, and then proceed to write metadata to D1 — which was not in dry-run scope. Real write, orphaned record. The fix: once any write tool in a run hits dry-run, propagate a flag for that runId that forces all subsequent writes in the same run to dry-run too. // after intercepting first dry-run write await ctx.env.KV.put dryrun active:${ctx.runId} , "1", { expirationTtl: 3600, } ; // every subsequent hook checks this flag const isDryRunActive = await ctx.env.KV.get dryrun active:${ctx.runId} === "1"; One more thing that burned me: if the hook itself fails — say, KV goes temporarily unavailable — Claude Code's default behavior is fall-through. The tool call executes anyway, dry-run flag ignored. Last week a KV spike caused hook timeouts and 3 agents wrote directly to production. No data loss because those ops were idempotent, but it was luck. Hook failure needs its own alert, separate from agent failure. I wrote up the full breakdown — including the dry-run propagation edge cases, R2 + D1 orphan scenarios, and where this pattern completely falls apart read-modify-write loops, APIs with side-effectful reads — over on riversealab.com.