Seal the First Five Commands Before Your On-Call Bot Invents a Restart

An engineer proposes a YAML-based runbook contract that seals the first five commands after an alert as read-only, preventing AI agents from inventing mutations like restarts during incidents. The design includes a freeze calendar and CI linting to enforce the contract, aiming to reduce human error under pressure.

The first five commands after a page should be sealed, read-only, and boring, because creativity at 3 a.m. is how incidents get worse. I want every alert to land on a contract, not a chat transcript that an agent can rewrite. Have you noticed how quickly an assistant jumps from checking latency to restarting a deployment under pressure? That jump is the real bug, and a linter can catch it before the next on-call rotation starts. Agentic tools are getting better at sounding like they belong in the incident channel. They also assume missing facts, which is the same failure people keep reporting when they wire agents to cloud APIs. Why should a paging bot invent kubectl rollout restart when your freeze calendar already forbids writes? Why should the first command after a page ever be a mutation? I treat the opening of a runbook as a sealed list, not as a prompt the model can keep editing. The model may draft the narrative, the diagrams, and the paragraph that explains why the alert exists. It does not get to author the first commands, the escalation timer, or the freeze switch. Those three fields are the contract, and they belong in YAML that a test can fail in CI. I keep four blocks, and I refuse to ship a runbook that is missing any of them. Does your current wiki page even list the first command as a string you can grep? If not, an assistant will happily fill the gap with something that sounds like expertise. The order matters more than the prose. Humans under a page will run whatever sits at the top, and models will imitate that habit. If the top is sealed and dull, the rest of the incident has a chance to stay honest. The following example is a proposal you can save as runbooks/checkout-latency.yaml and validate locally. It is not a production dump from a company, and I am not claiming it pages a real cluster. apiVersion: oncall.contract/v1 id: checkout-latency-p1 owner: sre-payments alert: name: CheckoutLatencyHigh severity: p1 page if: duration minutes gte: 5 error budget remaining pct lt: 25 ticket if: duration minutes lt: 5 first commands: sealed: true max steps: 5 mode: read only steps: - id: fc1 purpose: confirm the alert is still firing cmd: kubectl -n checkout get deploy checkout-api -o jsonpath='{.status.conditions}' timeout sec: 20 - id: fc2 purpose: check replica readiness without touching pods cmd: kubectl -n checkout get pods -l app=checkout-api -o wide timeout sec: 20 - id: fc3 purpose: read recent warning events only cmd: kubectl -n checkout get events --field-selector type=Warning --sort-by=.lastTimestamp timeout sec: 30 - id: fc4 purpose: sample current latency without mutating anything cmd: curl -fsS --max-time 5 http://127.0.0.1:9090/api/v1/query?query=up timeout sec: 10 - id: fc5 purpose: confirm freeze state before anyone talks about restarts cmd: python3 tools/freeze status.py --service checkout-api timeout sec: 10 escalation: after minutes: 10 to role: secondary-oncall require human: true stop if: alert resolved freeze: calendar: ./calendars/checkout-freeze.yaml during freeze: allow mutating commands: false page still allowed: true unfreeze: requires roles: incident-commander, sre-lead min approvers: 2 ttl minutes: 30 note: Unfreeze unlocks the mutating allowlist only, never the first commands seal. Look at first commands.sealed . That boolean is the whole point of the file. If a generator rewrites those steps, the linter should fail the pull request, not the incident. Would you merge a runbook that lets a chatbot swap get for delete because the prompt said be helpful? Pair the runbook with a calendar file. Empty rumors in Slack are not a freeze source of truth. calendars/checkout-freeze.yaml service: checkout-api timezone: UTC windows: - id: sep-freeze-2026 start: 2026-09-01T00:00:00Z end: 2026-09-08T00:00:00Z reason: payment cutover mutating allowlist when unfrozen: - kubectl -n checkout rollout restart deploy/checkout-api I do not want freeze to mean please be careful tonight. I want freeze to mean the mutating allowlist is empty until two humans spend a TTL. Unfreeze is an explicit event with two roles and a short clock, because a permanent unfreeze is just another wiki lie that nobody re-reads. | State | First commands | Mutating remediations | Page? | Who can change the lock | |---|---|---|---|---| | freeze | sealed, read-only | blocked | yes | two roles, TTL | | unfrozen | still sealed, read-only | allowlist only | yes | lock returns after TTL | | no calendar | fail closed | fail closed | ticket only | owner must add a calendar | Would you let an agent flip that table because the system prompt said be helpful? I would not, and neither should your pager. During freeze, the only honest remediation text is collect evidence, then wait for a named unfreeze. After a valid unfreeze, you may run a separate allowlisted command, never a rewritten first command. The seal does not thaw when the lock opens. That is the rule I want sitting next to every chatbot integration. Save this as tools/lint runbook.py . Treat it as a local check you can run in CI. I am not publishing pass or fail metrics from a fleet I do not have. bash /usr/bin/env python3 """Lint an on-call runbook contract. Proposal: run against YAML in ./runbooks.""" from future import annotations import sys from pathlib import Path try: import yaml except ImportError: print "Install pyyaml: pip install pyyaml", file=sys.stderr sys.exit 2 MUTATING TOKENS = " delete ", " apply ", " patch ", " scale ", " rollout restart", " drain ", " cordon ", " rm ", " drop ", READ VERBS = " get ", " describe ", " logs ", " top " def is read only cmd: str - bool: compact = " ".join cmd.split padded = f" {compact.lower } " if any tok in padded for tok in MUTATING TOKENS : return False if compact.startswith "kubectl" and not any v in f" {compact} " for v in READ VERBS : return False return compact.startswith "kubectl", "curl -fsS", "curl -sS", "python3 tools/freeze status.py" def lint doc: dict - list str : errors: list str = alert = doc.get "alert" or {} if not alert.get "name" or not alert.get "severity" : errors.append "alert.name and alert.severity are required" if not alert.get "page if" or alert.get "ticket if" : errors.append "alert must declare page if or ticket if" fc = doc.get "first commands" or {} if fc.get "sealed" is not True: errors.append "first commands.sealed must be true" if fc.get "mode" = "read only": errors.append "first commands.mode must be read only" steps = fc.get "steps" or max steps = int fc.get "max steps" or 5 if not 1 <= len steps <= max steps: errors.append "first commands.steps must contain 1..max steps entries" for step in steps: cmd = str step.get "cmd" or "" if not is read only cmd : errors.append f"mutating or unknown first command: {cmd}" if int step.get "timeout sec" or 0 <= 0: errors.append f"step {step.get 'id' } needs timeout sec" esc = doc.get "escalation" or {} if int esc.get "after minutes" or 0 <= 0: errors.append "escalation.after minutes must be 0" if not esc.get "require human" : errors.append "escalation.require human must be true" if not esc.get "to role" : errors.append "escalation.to role is required" freeze = doc.get "freeze" or {} if not freeze.get "calendar" : errors.append "freeze.calendar is required fail closed " during = freeze.get "during freeze" or {} if during.get "allow mutating commands" is not False: errors.append "during freeze.allow mutating commands must be false" unfreeze = freeze.get "unfreeze" or {} if int unfreeze.get "min approvers" or 0 < 2: errors.append "unfreeze.min approvers must be = 2" if int unfreeze.get "ttl minutes" or 0 <= 0: errors.append "unfreeze.ttl minutes must be 0" note = str unfreeze.get "note" or "" .lower if "never" not in note or "first commands" not in note: errors.append "unfreeze note must say the first commands seal never thaws" return errors def main argv: list str - int: if len argv = 2: print "usage: lint runbook.py <runbook.yaml ", file=sys.stderr return 2 path = Path argv 1 doc = yaml.safe load path.read text errors = lint doc if errors: print f"FAIL {path}" for item in errors: print f" - {item}" return 1 print f"PASS {path}" return 0 if name == " main ": raise SystemExit main sys.argv A matching stub keeps fc5 reproducible without pretending you already have a freeze service. bash /usr/bin/env python3 """Proposal stub: print freeze state from the calendar file.""" import argparse from datetime import datetime, timezone from pathlib import Path import yaml parser = argparse.ArgumentParser parser.add argument "--service", required=True parser.add argument "--calendar", default="calendars/checkout-freeze.yaml" args = parser.parse args doc = yaml.safe load Path args.calendar .read text now = datetime.now timezone.utc active = False for window in doc.get "windows" or : start = datetime.fromisoformat window "start" .replace "Z", "+00:00" end = datetime.fromisoformat window "end" .replace "Z", "+00:00" if start <= now < end and doc.get "service" == args.service: active = True print f"FREEZE active id={window 'id' } reason={window 'reason' }" if not active: print "FREEZE inactive" Run the happy path, then break it on purpose: pip install pyyaml python3 tools/lint runbook.py runbooks/checkout-latency.yaml expected: PASS runbooks/checkout-latency.yaml Copy the YAML, change fc1 to kubectl -n checkout rollout restart deploy/checkout-api , and run the linter again. If it does not exit 1, you do not have a seal. You have a suggestion that will fire during a real page. I still want help writing the why this alert exists section, because humans abandon runbooks that read like legal documents. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is enough for the drafting half of this workflow if you do not want that loop on a production laptop. The prompt I actually send is intentionally rude to the model, because polite prompts invite invented flags. You may rewrite purpose strings and the human-readable summary. You may not add, remove, or edit first commands.steps .cmd. You may not set sealed to false. You may not lower unfreeze.min approvers. You may not invent kubectl flags that are not already in the file. Return YAML only if lint runbook.py would still pass. If the model cannot obey that, I paste the prose back by hand and leave the commands untouched. Have you tried asking an agent to stop being helpful? It is a surprisingly good filter for on-call text. A tiny workflow that stays honest looks like this: lint runbook.py runs in CI on every pull request that touches runbooks/ .That is the whole loop. No dashboard screenshots, no invented latency wins, and no claim that the bot closed the incident. Skip this contract if you do not own the cluster, because a sealed kubectl list is still a credential problem. Skip it if your alerts are not mapped to a single service owner, because escalation to a role that does not exist is theater. Skip it if you need the bot to execute remediations unattended, since this design refuses that job on purpose. Also skip it if your freeze calendar is a rumor in chat. The linter fail-closes without a calendar path, and that will annoy you until you write the file. Good. Annoyance before the page is cheaper than a restart during a payment cutover. This linter does not prove a command is safe. It only proves the string looks read-only and the freeze block is present. A curl to the wrong internal URL can still be harmful, and a kubectl logs on a huge pod can still stall a laptop. Timeouts are a courtesy, not a sandbox, and they will not save you from a bad kubeconfig. I also cannot claim a specific model name, quota, or hardware profile for the drafting step, because those numbers go stale and I will not invent them. The value is the seal, not the vendor sticker. If you strip every product name out of this article, you should still have a YAML file, a calendar, and a Python exit code. One more uncomfortable question sits under all of this. If your runbook cannot fail a unit test, why would you let an agent touch it during a page?