The Runbook as Infrastructure

Dan Shapiro and Nate B. Jones have provided frameworks for AI agents operating in degraded systems, distinguishing human-facing runbooks from agent-facing runbooks by structure rather than detail. The concepts, detailed in Shapiro's blog and Jones's video, emphasize explicit decision points, rollback steps, and escalation criteria to prevent agents from making coherent but incorrect inferences.

Building the AI Dark Factory — Issue 18 I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word. Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding Why Most of You Won’t Make It Past Level 2 ” is what triggered this entire newsletter. natebjones.com https://www.natebjones.com/ — Watch the video https://youtu.be/bDcgHzCBgmQ This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 in context of the title of this newsletter on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair. Issue 17 built evals that intercept agent intent before execution. They answer the question: is this situation safe to proceed? Issue 18 addresses the adjacent problem. When the situation is already degraded and the agent must act — when the payment gateway is returning timeouts and orders are failing — the agent cannot wait for a pre-flight check. It must make decisions. Those decisions require the same kind of explicit structure that evals provide, applied to a context where the system is already broken. That is what a runbook is for. And the difference between a human-facing runbook and an agent-facing runbook is not detail or completeness. It is structure. Every place where a human runbook says "check if," "consider," "if needed," or "verify" is a place where an agent must infer. The inferences are not random. They are coherent. The agent applies the information it has and reaches a conclusion that follows from that information. The conclusion is wrong when the information is insufficient. Five dimensions where this plays out in practice. Decision points. Human version: "If the problem is widespread, consider escalating to the payment gateway provider." An agent has no named threshold for "widespread." It may observe that 2 of 100 orders are failing, conclude this is not widespread the majority succeed , and not escalate. Or it may observe any orders failing, conclude this is widespread, and escalate immediately. Both inferences are internally consistent. Both can be wrong. An agent that does not escalate at 30% failure rate allows inventory holds to accumulate silently — 15 minutes per PAYMENT PENDING order — until they expire and those orders are permanently lost. Rollback steps. Human version: "If you made configuration changes, revert them if the issue persists." Which changes? Which revert mechanism — git revert , git reset , restoring from backup? "Persists" according to what measurement? An agent that uses git reset --hard HEAD~1 instead of git revert HEAD --no-edit may discard uncommitted findings notes written during the investigation. In this project, the findings file is the incident record. Losing it means the escalation has no history. Escalation criteria. Human version: "Escalate if the issue persists or if you're unsure of the root cause." "If you're unsure" is not a condition an agent can evaluate. An agent does not have uncertainty — it has a model of the situation and it acts on that model. If the model is wrong, the agent is not unsure. It is wrong, and it does not know it is wrong. The only escalation triggers an agent can evaluate are observable states: specific response codes, specific test failures, specific time thresholds. Environment assumptions. Human runbooks never state which repository or environment they apply to — the human operator knows. An agent operating across multiple repositories executes runbook commands in whatever the current working directory is. A git revert HEAD --no-edit in the wrong repository reverts the most recent commit there — which may be unrelated to the payment gateway issue. The command succeeds. No error message. The wrong thing is reverted. Completion criteria. Human version: "Verify the service is functioning normally before closing the incident." An agent must choose what to verify. It might send a single successful request and close the incident. It might run behavioral tests but not contract tests. It might accept "all tests pass" as completion even when some tests were already failing before the incident. The verification is only meaningful if it is specific. Built for the payment gateway degraded scenario — the realistic failure case for this project. A good runbook, the kind a competent on-call engineer would write and follow. It contains the standard sections: overview, symptoms, investigation steps, mitigation, rollback, escalation, post-incident. It contains five places where an agent must infer. Here is the most dangerous one. Under mitigation, Step 3: "Consider adjusting the timeout configuration if the gateway's response time has increased significantly." This is the instruction that causes an agent to take a damaging action that a human operator would not take. The human operator knows — from experience, from reading the codebase, from understanding the stub design — that PAYMENT TIMEOUT SECONDS is the per-attempt HTTP client timeout, that the payment-timeout stub delays 6000ms to simulate a gateway that does not respond within 5 seconds, and that increasing PAYMENT TIMEOUT SECONDS above 6 seconds changes the code path from TimeoutException to response handling. The TimeoutException path produces PAYMENT PENDING with a 15-minute inventory hold. The response-handling path produces PAYMENT FAILED with the stub's 504 body. Different status. Different downstream behavior. Different customer experience. The agent reads: "the timeout is 5 seconds, the gateway is taking 6 seconds, the runbook says to adjust the timeout." It sets PAYMENT TIMEOUT SECONDS = 7 . The action is logical. It follows from the available information. It is wrong. And it does not produce an error during execution. The configuration change applies. The service restarts. The gateway continues responding slowly. The agent checks whether the issue is resolved and — depending on what it checks — may conclude that the change helped. The damage to Scenario 5 is invisible until the timeout scenario is specifically exercised. Seven sections. All required. Section 1: Pre-flight environment check. Before any action: confirm the correct repository check git remote get-url origin , expected to contain "lvl5engineer-order-api" . Confirm the test suite baseline run full Gherkin suite, document which tests pass before any intervention — you cannot distinguish your changes from pre-existing failures without this . Confirm the current state of can i deploy.py . The baseline is the reference point for every subsequent step. Section 2: Symptom identification. Not "investigate the gateway issue." A decision tree with four named branches, each producing a specific next action: PAYMENT FAILED with decline reason: INSUFFICIENT FUNDS → customer issue, not gateway issue, no mitigation required PAYMENT PENDING with retry count: 2 → gateway timing out, proceed to Section 3 order id → gateway unreachable, proceed to Section 4 status: CONFIRMED but no transaction id → Pact contract violation, run Pact tests, halt and escalateEach branch is an observable state. No branch requires the agent to assess whether the situation is "widespread" or "significant." Section 3: Gateway timeout mitigation. Step 1 checks the current PAYMENT TIMEOUT SECONDS . Step 2 checks the current MAX PAYMENT RETRIES . Step 3 is the decision: IF the payment gateway's documented SLA timeout is greater than the current PAYMENT TIMEOUT SECONDS: → Document the proposed change. Run the operation scope eval. Make the change. Run the full test suite. If any test fails: revert immediately. IF the gateway's documented SLA timeout is less than or equal to the current PAYMENT TIMEOUT SECONDS: → The gateway is genuinely degraded beyond its SLA. Do not increase the timeout. Proceed to Section 4. The SLA timeout must come from gateway documentation — not from the stub file. This is the instruction that prevents the PAYMENT TIMEOUT SECONDS = 7 failure. The stub delay fixedDelayMilliseconds: 6000 is a simulation of timeout behavior, not the gateway's actual SLA. An agent that uses the stub delay as the SLA threshold would conclude that 6000ms 5000ms means the client is too aggressive. It would be right about the comparison and wrong about the meaning. Step 4: verify with the specific command that exercises the actual timeout code path: pytest "tests/steps/test order creation.py::test order handling is graceful when the payment gateway times out" -v Not pytest tests/steps/ -v . Not pytest -k timeout . The specific test, by full path. This was not the original runbook command — the dry run found it. Section 4: Gateway unavailable mitigation. Probe commands to confirm unreachability versus misconfiguration. Note: the mock server on port 8091 only runs inside an active pytest session — probing outside a test session always shows "unreachable" regardless of configuration, which the runbook anticipates. Section 5: Rollback. git log --oneline -5 , identify the change commit, git revert hash --no-edit , run the full suite. If the revert produces test failures: halt. Do not attempt further changes without human review. Section 6: Escalation criteria. Named conditions that trigger escalation — not "if you're unsure": Escalation means: write a findings entry with current state, steps taken, and exact output of each step. Then stop. Section 7: Completion criteria. Five named checks. All five must be true for the runbook to be complete: pytest tests/steps/ -v → all pass pytest tests/pact/ -v → all pass scripts/can i deploy.py → "ALL CONTRACTS VERIFIED" PAYMENT TIMEOUT SECONDS and MAX PAYMENT RETRIES documented with current values ops: prefixIf any criterion is not met: the runbook is not complete. The agent-facing runbook was executed against the current project state with the payment-timeout stub active — the realistic test of whether the runbook's commands produce the expected outputs. Pre-flight check: repository confirmed, baseline documented 11 Gherkin scenarios passing, 4 Pact interactions passing, can-i-deploy green . Symptom identification: the timeout stub produces orders with status: PAYMENT PENDING , retry count: 2 . Correct branch: Section 3. Section 3 reached the SLA documentation requirement. In the test environment, no external gateway documentation exists. The stub delay serves as a proxy but using it would require understanding that the stub delay must exceed the client timeout to trigger TimeoutException . Decision documented: SLA unavailable from documentation; no timeout change made. The runbook handled this correctly — it required documentation, found it absent, and stopped rather than inferring. The gap the dry run found: The original runbook verification command was pytest tests/steps/test order creation.py -v -k timeout . Actual output: collected 5 items / 5 deselected / 0 selected exit code 5 — no tests selected The -k timeout keyword does not match test order handling is graceful when the payment gateway times out because the test name uses "times out" not "timeout." Exit code 5 is ambiguous — a runbook reader might interpret it as "no timeout tests exist" or "the test framework is broken." The runbook was updated to use the full test path. After the fix: 1 passed in 11.83s . A runbook that is written but never run is a runbook whose commands have never been validated. This is the same discipline test maintenance requires: execute it, find the gaps, apply the fixes before they are needed. All five completion criteria met. Dry run complete. The evals from Issue 17 prevent the PAYMENT TIMEOUT SECONDS = 7 change from being made in a normal session — the operation scope eval's Q1 would check ADR-001, and the decision index would surface the relevant documentation. In a degraded state, the eval sequence assumes a functioning pre-flight process. The runbook operates in the gap where pre-flight assumptions no longer hold — where the service is already broken and the agent must diagnose and act without the normal session structure. The eval answers: is this situation safe to proceed? The runbook answers: the situation is already unsafe — here is how to make it safe again, step by step, with no gaps for inference. Both are necessary. Neither is sufficient without the other. Layer 3 is the three artifacts working together: evals prevent damage before it happens, ADRs capture the decisions that explain why, and runbooks provide the explicit structure for the cases where damage is already in progress. Next issue: The Full Stack — building a complete new feature with all three layers in simultaneous use, and comparing the result to Issue 3 when the agent had only a spec. Sources & Further Reading This article was written with the assistance of AI tools.