# The Runbook as Infrastructure > Source: > Published: 2026-08-17 13:30:00+00:00 *Building the AI Dark Factory — Issue #18* I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word. Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. [danshapiro.com](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory) Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. [natebjones.com](https://www.natebjones.com/) — [Watch the video](https://youtu.be/bDcgHzCBgmQ) This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair. Issue #17 built evals that intercept agent intent before execution. They answer the question: is this situation safe to proceed? Issue #18 addresses the adjacent problem. When the situation is already degraded and the agent must act — when the payment gateway is returning timeouts and orders are failing — the agent cannot wait for a pre-flight check. It must make decisions. Those decisions require the same kind of explicit structure that evals provide, applied to a context where the system is already broken. That is what a runbook is for. And the difference between a human-facing runbook and an agent-facing runbook is not detail or completeness. It is structure. Every place where a human runbook says "check if," "consider," "if needed," or "verify" is a place where an agent must infer. The inferences are not random. They are coherent. The agent applies the information it has and reaches a conclusion that follows from that information. The conclusion is wrong when the information is insufficient. Five dimensions where this plays out in practice. **Decision points.** Human version: "If the problem is widespread, consider escalating to the payment gateway provider." An agent has no named threshold for "widespread." It may observe that 2 of 100 orders are failing, conclude this is not widespread (the majority succeed), and not escalate. Or it may observe any orders failing, conclude this is widespread, and escalate immediately. Both inferences are internally consistent. Both can be wrong. An agent that does not escalate at 30% failure rate allows inventory holds to accumulate silently — 15 minutes per PAYMENT_PENDING order — until they expire and those orders are permanently lost. **Rollback steps.** Human version: "If you made configuration changes, revert them if the issue persists." Which changes? Which revert mechanism — `git revert` , `git reset` , restoring from backup? "Persists" according to what measurement? An agent that uses `git reset --hard HEAD~1` instead of `git revert HEAD --no-edit` may discard uncommitted findings notes written during the investigation. In this project, the findings file is the incident record. Losing it means the escalation has no history. **Escalation criteria.** Human version: "Escalate if the issue persists or if you're unsure of the root cause." "If you're unsure" is not a condition an agent can evaluate. An agent does not have uncertainty — it has a model of the situation and it acts on that model. If the model is wrong, the agent is not unsure. It is wrong, and it does not know it is wrong. The only escalation triggers an agent can evaluate are observable states: specific response codes, specific test failures, specific time thresholds. **Environment assumptions.** Human runbooks never state which repository or environment they apply to — the human operator knows. An agent operating across multiple repositories executes runbook commands in whatever the current working directory is. A `git revert HEAD --no-edit` in the wrong repository reverts the most recent commit there — which may be unrelated to the payment gateway issue. The command succeeds. No error message. The wrong thing is reverted. **Completion criteria.** Human version: "Verify the service is functioning normally before closing the incident." An agent must choose what to verify. It might send a single successful request and close the incident. It might run behavioral tests but not contract tests. It might accept "all tests pass" as completion even when some tests were already failing before the incident. The verification is only meaningful if it is specific. Built for the payment gateway degraded scenario — the realistic failure case for this project. A good runbook, the kind a competent on-call engineer would write and follow. It contains the standard sections: overview, symptoms, investigation steps, mitigation, rollback, escalation, post-incident. It contains five places where an agent must infer. Here is the most dangerous one. Under mitigation, Step 3: "Consider adjusting the timeout configuration if the gateway's response time has increased significantly." This is the instruction that causes an agent to take a damaging action that a human operator would not take. The human operator knows — from experience, from reading the codebase, from understanding the stub design — that `PAYMENT_TIMEOUT_SECONDS` is the per-attempt HTTP client timeout, that the payment-timeout stub delays 6000ms to simulate a gateway that does not respond within 5 seconds, and that increasing `PAYMENT_TIMEOUT_SECONDS` above 6 seconds changes the code path from `TimeoutException` to response handling. The `TimeoutException` path produces `PAYMENT_PENDING` with a 15-minute inventory hold. The response-handling path produces `PAYMENT_FAILED` with the stub's 504 body. Different status. Different downstream behavior. Different customer experience. The agent reads: "the timeout is 5 seconds, the gateway is taking 6 seconds, the runbook says to adjust the timeout." It sets `PAYMENT_TIMEOUT_SECONDS = 7` . The action is logical. It follows from the available information. It is wrong. And it does not produce an error during execution. The configuration change applies. The service restarts. The gateway continues responding slowly. The agent checks whether the issue is resolved and — depending on what it checks — may conclude that the change helped. The damage to Scenario 5 is invisible until the timeout scenario is specifically exercised. Seven sections. All required. **Section 1: Pre-flight environment check.** Before any action: confirm the correct repository (check `git remote get-url origin` , expected to contain "lvl5engineer-order-api"). Confirm the test suite baseline (run full Gherkin suite, document which tests pass before any intervention — you cannot distinguish your changes from pre-existing failures without this). Confirm the current state of `can_i_deploy.py` . The baseline is the reference point for every subsequent step. **Section 2: Symptom identification.** Not "investigate the gateway issue." A decision tree with four named branches, each producing a specific next action: `PAYMENT_FAILED` with `decline_reason: INSUFFICIENT_FUNDS` → customer issue, not gateway issue, no mitigation required`PAYMENT_PENDING` with `retry_count: 2` → gateway timing out, proceed to Section 3`order_id` → gateway unreachable, proceed to Section 4`status: CONFIRMED` but no `transaction_id` → Pact contract violation, run Pact tests, halt and escalateEach branch is an observable state. No branch requires the agent to assess whether the situation is "widespread" or "significant." **Section 3: Gateway timeout mitigation.** Step 1 checks the current `PAYMENT_TIMEOUT_SECONDS` . Step 2 checks the current `MAX_PAYMENT_RETRIES` . Step 3 is the decision: ``` IF the payment gateway's documented SLA timeout is greater than the current PAYMENT_TIMEOUT_SECONDS: → Document the proposed change. Run the operation scope eval. Make the change. Run the full test suite. If any test fails: revert immediately. IF the gateway's documented SLA timeout is less than or equal to the current PAYMENT_TIMEOUT_SECONDS: → The gateway is genuinely degraded beyond its SLA. Do not increase the timeout. Proceed to Section 4. ``` The SLA timeout must come from gateway documentation — not from the stub file. This is the instruction that prevents the `PAYMENT_TIMEOUT_SECONDS = 7` failure. The stub delay (`fixedDelayMilliseconds: 6000` ) is a simulation of timeout behavior, not the gateway's actual SLA. An agent that uses the stub delay as the SLA threshold would conclude that 6000ms > 5000ms means the client is too aggressive. It would be right about the comparison and wrong about the meaning. Step 4: verify with the specific command that exercises the actual timeout code path: ``` pytest "tests/steps/test_order_creation.py::test_order_handling_is_graceful_when_the_payment_gateway_times_out" -v ``` Not `pytest tests/steps/ -v` . Not `pytest -k timeout` . The specific test, by full path. This was not the original runbook command — the dry run found it. **Section 4: Gateway unavailable mitigation.** Probe commands to confirm unreachability versus misconfiguration. Note: the mock server on port 8091 only runs inside an active pytest session — probing outside a test session always shows "unreachable" regardless of configuration, which the runbook anticipates. **Section 5: Rollback.** `git log --oneline -5` , identify the change commit, `git revert [hash] --no-edit` , run the full suite. If the revert produces test failures: halt. Do not attempt further changes without human review. **Section 6: Escalation criteria.** Named conditions that trigger escalation — not "if you're unsure": Escalation means: write a findings entry with current state, steps taken, and exact output of each step. Then stop. **Section 7: Completion criteria.** Five named checks. All five must be true for the runbook to be complete: `pytest tests/steps/ -v` → all pass`pytest tests/pact/ -v` → all pass`scripts/can_i_deploy.py` → "ALL CONTRACTS VERIFIED"`PAYMENT_TIMEOUT_SECONDS` and `MAX_PAYMENT_RETRIES` documented with current values`ops:` prefixIf any criterion is not met: the runbook is not complete. The agent-facing runbook was executed against the current project state with the payment-timeout stub active — the realistic test of whether the runbook's commands produce the expected outputs. Pre-flight check: repository confirmed, baseline documented (11 Gherkin scenarios passing, 4 Pact interactions passing, can-i-deploy green). Symptom identification: the timeout stub produces orders with `status: PAYMENT_PENDING` , `retry_count: 2` . Correct branch: Section 3. Section 3 reached the SLA documentation requirement. In the test environment, no external gateway documentation exists. The stub delay serves as a proxy but using it would require understanding that the stub delay must exceed the client timeout to trigger `TimeoutException` . Decision documented: SLA unavailable from documentation; no timeout change made. The runbook handled this correctly — it required documentation, found it absent, and stopped rather than inferring. **The gap the dry run found:** The original runbook verification command was `pytest tests/steps/test_order_creation.py -v -k timeout` . Actual output: ``` collected 5 items / 5 deselected / 0 selected (exit code 5 — no tests selected) ``` The `-k timeout` keyword does not match `test_order_handling_is_graceful_when_the_payment_gateway_times_out` because the test name uses "times_out" not "timeout." Exit code 5 is ambiguous — a runbook reader might interpret it as "no timeout tests exist" or "the test framework is broken." The runbook was updated to use the full test path. After the fix: `1 passed in 11.83s` . A runbook that is written but never run is a runbook whose commands have never been validated. This is the same discipline test maintenance requires: execute it, find the gaps, apply the fixes before they are needed. All five completion criteria met. Dry run complete. The evals from Issue #17 prevent the `PAYMENT_TIMEOUT_SECONDS = 7` change from being made in a normal session — the operation scope eval's Q1 would check ADR-001, and the decision index would surface the relevant documentation. In a degraded state, the eval sequence assumes a functioning pre-flight process. The runbook operates in the gap where pre-flight assumptions no longer hold — where the service is already broken and the agent must diagnose and act without the normal session structure. The eval answers: is this situation safe to proceed? The runbook answers: the situation is already unsafe — here is how to make it safe again, step by step, with no gaps for inference. Both are necessary. Neither is sufficient without the other. Layer 3 is the three artifacts working together: evals prevent damage before it happens, ADRs capture the decisions that explain why, and runbooks provide the explicit structure for the cases where damage is already in progress. *Next issue: The Full Stack — building a complete new feature with all three layers in simultaneous use, and comparing the result to Issue #3 when the agent had only a spec.* **Sources & Further Reading** *This article was written with the assistance of AI tools.*