Runbooks Are Dead; Make Operations Executable

A blog post argues that traditional prose runbooks are an architectural dead end in the AI era because AI agents cannot read Grafana screenshots, SSH into bastion hosts, or run scripts from personal home directories, and should instead be replaced by config-driven internal services with APIs that accept structured inputs and return structured results. The author describes a four-stage progression from prose runbooks to personal scripts to shared tools to config-driven services, and recounts a database failover incident that took 25 minutes and required remembering 10 steps. The post contends that executable runbooks let AI agents, Slack bots, and CLIs invoke operations without knowing internal implementation details.

Runbooks Are Dead; Make Operations Executable The best runbook I ever read was also the most dangerous one I ever followed . It was about database failover procedure. 14 steps. Step 3 said “verify the replica is caught up” but didn’t say how. Step 7 said “if the promotion fails, escalate to the DBA team” like the DBA team was sitting around at 3 AM. Step 11 was just the word “validate” followed by a period. The whole document was a confession. It said: “We know this could go wrong, someone wrote down what they remembered, and we hope you figure out the rest”. This is the standard state of operational knowledge at most companies and in the AI era it’s an architectural dead end . The runbook is the institutional debt Every runbook starts with good intentions : something breaks, someone fixes it and they write down the steps so the next person doesn’t suffer. Six months later, half the commands are deprecated, the dashboard URLs point to a monitoring tool that was replaced, and the person who wrote it left the company. Think about what a runbook really asks of the person on call. At 3 AM, with adrenaline pumping and Slack pinging, you’re supposed to read a wall of text, mentally parse conditional branches “if the primary is unreachable but the replica is healthy, proceed to step 4, otherwise skip to step 8” , manually copy-paste commands into a terminal, and hope you don’t typo a hostname. Let’s be real: nobody practices runbooks and nobody rehearses them. Your first time through the procedure is the incident itself. This looks like a runbook, but it behaves like a liability with good formatting . Knowledge should be executable Here’s the shift: stop asking where to document the procedure, and start asking how to build it into the platform. An executable runbook behaves like a service: you send it inputs, it runs the steps, and it returns a result. The real logic lives in tested, versioned code. The progression usually goes like this: - Stage 1: The prose runbook. A page someone wrote after the last outage. Accurate for about two weeks, then slowly rotting. - Stage 2: The personal script. Someone on the team writes a bash script that automates the worst parts. It lives in their home directory on a bastion host. Nobody else knows it exists. When they leave, it’s gone. - Stage 3: The shared tool. The script gets checked into a repo. It has a README. Maybe a Makefile. People use it, but they have to know where the repo is, clone it, install dependencies, and run it from the right machine. - Stage 4: The config-driven service. The tool becomes a proper internal service with an API. It reads configuration from a central source of truth. It validates inputs before acting. It logs what it did. It can be called by a Slack bot, a CLI, or an AI agent without any of them knowing how it works internally. - Stage 4 is where executable runbooks unlock the AI era . Here’s the hard part: AI agents can’t always follow a runbook. They can’t read a Grafana screenshot, SSH into a bastion host, or run a script from someone’s home directory. What they can do is call APIs, invoke tools, and pass structured inputs to services that return structured results. During incidents, the system should do the procedure for you Let me make this concrete. Today . Today you get paged: the primary database is down. You open the internal wiki and start following the steps. Check replication lag. Copy the command. Swap in the hostname. Run it. Lag is zero. Good. Then you try to promote the replica, but it fails because you’re on the wrong jump host. So you find the right one, SSH in, run it again, and it works. You update the application connection string, forget the read-replica config, and suddenly half the services start failing because they’re still pointed at the old primary. You fix that too. Total time: 25 minutes. Steps you had to remember: 10. Things that went wrong because the runbook was out of date: 3. After . Same scenario. You get paged. You type /db-failover in Slack. The bot asks which cluster. You say orders-db-prod . It checks replication status, promotes the replica, updates the connection strings in the service mesh config, verifies all services can reach the new primary, and posts the result back to the incident channel. Total time: 90 seconds. Steps you had to remember: 1. Things that went wrong: 0. The executable version isn’t magic. Under the hood it’s calling the same commands. The difference is that the knowledge of how to do those steps, in what order , with what validation between each one , is encoded in a service rather than a document that someone reads for the first time during an outage. What makes it a tool instead of a script The real difference is whether the thing can be safely used at 3 AM by someone seeing it for the first time. - Every step validates before it acts . The tool checks replication health before touching anything. If active writes are still in flight, it aborts instead of corrupting data. The human doesn’t have to remember the preconditions. The tool refuses to proceed without them. - Every step has a failure path . Promotion fails? Roll back. Connectivity check fails? Report which services are broken. The tool handles the branches. The person on call only needs to know one command. - It’s idempotent . Running it twice on an already-promoted replica is a no-op, not a disaster. That matters at 3 AM when you’re not sure if the first attempt went through. - One endpoint, every consumer. POST /tools/db-failover is what the Slack bot calls, what the CLI calls, and what the MCP server exposes. One backend. One source of truth. Every surface gets the same capability. Why this matters for AI agents This is the part I care about most. The current excitement around AI operations assistants, AI SRE copilots, AI incident responders, all of it hits the same wall: these systems can only operate on structured, machine-accessible interfaces . A document can be read and summarized, but it still gives an AI agent nothing to call. When you turn a runbook into a tool, you’re helping more than the person on call. You’re building something your AI agents can use too. Today it might run from a Slack command. Tomorrow it might be called by an MCP server or an agent. Either way, it just does the job. This is the bridge between the other articles in this series. MCP servers need tools to expose https://shiftmag.dev/mcps-arent-apis-stop-treating-them-like-one-11420/ . Slack bots need tools to call https://shiftmag.dev/you-should-bring-your-ops-to-slack-11860/ article two . Executable runbooks are those tools. They’re the operational layer that both the human interface and the AI interface sit on top of. This has never been easier Five years ago, converting a runbook into an API-backed service meant writing a backend, wiring up auth, deploying it somewhere, and maintaining it forever. That’s why most teams never did it, the friction was too high. That equation has changed. Low-code and no-code automation tools like n8n have made it trivial to wrap existing APIs, databases, and CLI commands into callable workflows. You can build a db-failover tool in an afternoon by chaining HTTP requests, database queries, and conditional logic in a visual editor. Expose it as a webhook, and suddenly your Slack bot, your MCP server, and your AI agent all have something to call. The barrier isn’t technical anymore . We’re still defaulting to static documentation because that’s what we’ve always done. But when an n8n workflow can be built, tested, and deployed faster than you can format a runbook, “we don’t have time to automate it” stops being a valid excuse. Start with the procedure that scares you the most You don’t need to convert every runbook overnight. Start with the scariest runbook. Build the tool, test it in staging, use it in a maintenance window, then move on to the next one. Each tool you build reduces operational risk by a measurable amount . More importantly, each tool becomes a building block. Once db-failover exists as a tool, the full-site-recovery runbook doesn’t need to include database steps. It just calls db-failover and moves on. The end state is a platform where operational knowledge doesn’t live in documents but in services . Not because documents are bad, but because services are callable, testable, composable, and reachable by both humans and machines. | Principle | Why | |---|---| | Prose runbooks are institutional debt | They rot, require human interpretation under stress, and block AI integration entirely | | Knowledge should be executable | Encode procedures in services that validate, log, and run consistently every time | | Expose tools through one API, not scripts scattered across repos | A single backend serves Slack commands, CLI calls, and AI agent invocations | | Build for composition | Each tool should be independently callable so higher-level runbooks can chain them | | Validate at every step, abort on failure | The tool enforces preconditions. Humans under stress forget them | | Start with the scariest procedure | Convert the runbook that causes the most anxiety first. Each tool reduces measurable risk | | Low-code tools eliminate the friction | n8n and similar platforms let you build callable workflows in hours instead of weeks. “No time to automate” is no longer valid | Humans still belong in operations; the machine should handle the repeatable work. Write the runbook once as code, and let the platform carry it.