The first five commands after a page should be sealed, read-only, and boring, because creativity at 3 a.m. is how incidents get worse. I want every alert to land on a contract, not a chat transcript that an agent can rewrite. Have you noticed how quickly an assistant jumps from checking latency to restarting a deployment under pressure? That jump is the real bug, and a linter can catch it before the next on-call rotation starts.
Agentic tools are getting better at sounding like they belong in the incident channel. They also assume missing facts, which is the same failure people keep reporting when they wire agents to cloud APIs. Why should a paging bot invent kubectl rollout restart
when your freeze calendar already forbids writes? Why should the first command after a page ever be a mutation?
I treat the opening of a runbook as a sealed list, not as a prompt the model can keep editing. The model may draft the narrative, the diagrams, and the paragraph that explains why the alert exists. It does not get to author the first commands, the escalation timer, or the freeze switch. Those three fields are the contract, and they belong in YAML that a test can fail in CI.
I keep four blocks, and I refuse to ship a runbook that is missing any of them. Does your current wiki page even list the first command as a string you can grep? If not, an assistant will happily fill the gap with something that sounds like expertise.
The order matters more than the prose. Humans under a page will run whatever sits at the top, and models will imitate that habit. If the top is sealed and dull, the rest of the incident has a chance to stay honest.
The following example is a proposal you can save as runbooks/checkout-latency.yaml
and validate locally. It is not a production dump from a company, and I am not claiming it pages a real cluster.
apiVersion: oncall.contract/v1
id: checkout-latency-p1
owner: sre-payments
alert:
name: CheckoutLatencyHigh
severity: p1
page_if:
duration_minutes_gte: 5
error_budget_remaining_pct_lt: 25
ticket_if:
duration_minutes_lt: 5
first_commands:
sealed: true
max_steps: 5
mode: read_only
steps:
- id: fc1
purpose: confirm the alert is still firing
cmd: kubectl -n checkout get deploy checkout-api -o jsonpath='{.status.conditions}'
timeout_sec: 20
- id: fc2
purpose: check replica readiness without touching pods
cmd: kubectl -n checkout get pods -l app=checkout-api -o wide
timeout_sec: 20
- id: fc3
purpose: read recent warning events only
cmd: kubectl -n checkout get events --field-selector type=Warning --sort-by=.lastTimestamp
timeout_sec: 30
- id: fc4
purpose: sample current latency without mutating anything
cmd: curl -fsS --max-time 5 http://127.0.0.1:9090/api/v1/query?query=up
timeout_sec: 10
- id: fc5
purpose: confirm freeze state before anyone talks about restarts
cmd: python3 tools/freeze_status.py --service checkout-api
timeout_sec: 10
escalation:
after_minutes: 10
to_role: secondary-oncall
require_human: true
stop_if: alert_resolved
freeze:
calendar: ./calendars/checkout-freeze.yaml
during_freeze:
allow_mutating_commands: false
page_still_allowed: true
unfreeze:
requires_roles: [incident-commander, sre-lead]
min_approvers: 2
ttl_minutes: 30
note: Unfreeze unlocks the mutating allowlist only, never the first_commands seal.
Look at first_commands.sealed
. That boolean is the whole point of the file. If a generator rewrites those steps, the linter should fail the pull request, not the incident. Would you merge a runbook that lets a chatbot swap get
for delete
because the prompt said be helpful?
Pair the runbook with a calendar file. Empty rumors in Slack are not a freeze source of truth.
service: checkout-api
timezone: UTC
windows:
- id: sep-freeze-2026
start: 2026-09-01T00:00:00Z
end: 2026-09-08T00:00:00Z
reason: payment cutover
mutating_allowlist_when_unfrozen:
- kubectl -n checkout rollout restart deploy/checkout-api
I do not want freeze to mean please be careful tonight. I want freeze to mean the mutating allowlist is empty until two humans spend a TTL. Unfreeze is an explicit event with two roles and a short clock, because a permanent unfreeze is just another wiki lie that nobody re-reads.
| State | First commands | Mutating remediations | Page? | Who can change the lock |
|---|---|---|---|---|
| freeze | sealed, read-only | blocked | yes | two roles, TTL |
| unfrozen | still sealed, read-only | allowlist only | yes | lock returns after TTL |
| no calendar | fail closed | fail closed | ticket only | owner must add a calendar |
Would you let an agent flip that table because the system prompt said be helpful? I would not, and neither should your pager. During freeze, the only honest remediation text is collect evidence, then wait for a named unfreeze. After a valid unfreeze, you may run a separate allowlisted command, never a rewritten first command. The seal does not thaw when the lock opens. That is the rule I want sitting next to every chatbot integration.
Save this as tools/lint_runbook.py
. Treat it as a local check you can run in CI. I am not publishing pass or fail metrics from a fleet I do not have.
#!/usr/bin/env python3
"""Lint an on-call runbook contract. Proposal: run against YAML in ./runbooks."""
from __future__ import annotations
import sys
from pathlib import Path
try:
import yaml
except ImportError:
print("Install pyyaml: pip install pyyaml", file=sys.stderr)
sys.exit(2)
MUTATING_TOKENS = (
" delete ",
" apply ",
" patch ",
" scale ",
" rollout restart",
" drain ",
" cordon ",
" rm ",
" drop ",
)
READ_VERBS = (" get ", " describe ", " logs ", " top ")
def is_read_only(cmd: str) -> bool:
compact = " ".join(cmd.split())
padded = f" {compact.lower()} "
if any(tok in padded for tok in MUTATING_TOKENS):
return False
if compact.startswith("kubectl") and not any(v in f" {compact} " for v in READ_VERBS):
return False
return compact.startswith(("kubectl", "curl -fsS", "curl -sS", "python3 tools/freeze_status.py"))
def lint(doc: dict) -> list[str]:
errors: list[str] = []
alert = doc.get("alert") or {}
if not alert.get("name") or not alert.get("severity"):
errors.append("alert.name and alert.severity are required")
if not (alert.get("page_if") or alert.get("ticket_if")):
errors.append("alert must declare page_if or ticket_if")
fc = doc.get("first_commands") or {}
if fc.get("sealed") is not True:
errors.append("first_commands.sealed must be true")
if fc.get("mode") != "read_only":
errors.append("first_commands.mode must be read_only")
steps = fc.get("steps") or []
max_steps = int(fc.get("max_steps") or 5)
if not 1 <= len(steps) <= max_steps:
errors.append("first_commands.steps must contain 1..max_steps entries")
for step in steps:
cmd = str(step.get("cmd") or "")
if not is_read_only(cmd):
errors.append(f"mutating or unknown first command: {cmd}")
if int(step.get("timeout_sec") or 0) <= 0:
errors.append(f"step {step.get('id')} needs timeout_sec")
esc = doc.get("escalation") or {}
if int(esc.get("after_minutes") or 0) <= 0:
errors.append("escalation.after_minutes must be > 0")
if not esc.get("require_human"):
errors.append("escalation.require_human must be true")
if not esc.get("to_role"):
errors.append("escalation.to_role is required")
freeze = doc.get("freeze") or {}
if not freeze.get("calendar"):
errors.append("freeze.calendar is required (fail closed)")
during = freeze.get("during_freeze") or {}
if during.get("allow_mutating_commands") is not False:
errors.append("during_freeze.allow_mutating_commands must be false")
unfreeze = freeze.get("unfreeze") or {}
if int(unfreeze.get("min_approvers") or 0) < 2:
errors.append("unfreeze.min_approvers must be >= 2")
if int(unfreeze.get("ttl_minutes") or 0) <= 0:
errors.append("unfreeze.ttl_minutes must be > 0")
note = str(unfreeze.get("note") or "").lower()
if "never" not in note or "first_commands" not in note:
errors.append("unfreeze note must say the first_commands seal never thaws")
return errors
def main(argv: list[str]) -> int:
if len(argv) != 2:
print("usage: lint_runbook.py <runbook.yaml>", file=sys.stderr)
return 2
path = Path(argv[1])
doc = yaml.safe_load(path.read_text())
errors = lint(doc)
if errors:
print(f"FAIL {path}")
for item in errors:
print(f" - {item}")
return 1
print(f"PASS {path}")
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
A matching stub keeps fc5
reproducible without pretending you already have a freeze service.
#!/usr/bin/env python3
"""Proposal stub: print freeze state from the calendar file."""
import argparse
from datetime import datetime, timezone
from pathlib import Path
import yaml
parser = argparse.ArgumentParser()
parser.add_argument("--service", required=True)
parser.add_argument("--calendar", default="calendars/checkout-freeze.yaml")
args = parser.parse_args()
doc = yaml.safe_load(Path(args.calendar).read_text())
now = datetime.now(timezone.utc)
active = False
for window in doc.get("windows") or []:
start = datetime.fromisoformat(window["start"].replace("Z", "+00:00"))
end = datetime.fromisoformat(window["end"].replace("Z", "+00:00"))
if start <= now < end and doc.get("service") == args.service:
active = True
print(f"FREEZE active id={window['id']} reason={window['reason']}")
if not active:
print("FREEZE inactive")
Run the happy path, then break it on purpose:
pip install pyyaml
python3 tools/lint_runbook.py runbooks/checkout-latency.yaml
Copy the YAML, change fc1
to kubectl -n checkout rollout restart deploy/checkout-api
, and run the linter again. If it does not exit 1, you do not have a seal. You have a suggestion that will fire during a real page.
I still want help writing the why this alert exists section, because humans abandon runbooks that read like legal documents. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is enough for the drafting half of this workflow if you do not want that loop on a production laptop.
The prompt I actually send is intentionally rude to the model, because polite prompts invite invented flags.
You may rewrite purpose strings and the human-readable summary.
You may not add, remove, or edit first_commands.steps[].cmd.
You may not set sealed to false.
You may not lower unfreeze.min_approvers.
You may not invent kubectl flags that are not already in the file.
Return YAML only if lint_runbook.py would still pass.
If the model cannot obey that, I paste the prose back by hand and leave the commands untouched. Have you tried asking an agent to stop being helpful? It is a surprisingly good filter for on-call text.
A tiny workflow that stays honest looks like this:
lint_runbook.py
runs in CI on every pull request that touches runbooks/
.That is the whole loop. No dashboard screenshots, no invented latency wins, and no claim that the bot closed the incident.
Skip this contract if you do not own the cluster, because a sealed kubectl
list is still a credential problem. Skip it if your alerts are not mapped to a single service owner, because escalation to a role that does not exist is theater. Skip it if you need the bot to execute remediations unattended, since this design refuses that job on purpose.
Also skip it if your freeze calendar is a rumor in chat. The linter fail-closes without a calendar path, and that will annoy you until you write the file. Good. Annoyance before the page is cheaper than a restart during a payment cutover.
This linter does not prove a command is safe. It only proves the string looks read-only and the freeze block is present. A curl
to the wrong internal URL can still be harmful, and a kubectl logs
on a huge pod can still stall a laptop. Timeouts are a courtesy, not a sandbox, and they will not save you from a bad kubeconfig.
I also cannot claim a specific model name, quota, or hardware profile for the drafting step, because those numbers go stale and I will not invent them. The value is the seal, not the vendor sticker. If you strip every product name out of this article, you should still have a YAML file, a calendar, and a Python exit code.
One more uncomfortable question sits under all of this. If your runbook cannot fail a unit test, why would you let an agent touch it during a page?