{"slug": "seal-the-first-five-commands-before-your-on-call-bot-invents-a-restart", "title": "Seal the First Five Commands Before Your On-Call Bot Invents a Restart", "summary": "An engineer proposes a YAML-based runbook contract that seals the first five commands after an alert as read-only, preventing AI agents from inventing mutations like restarts during incidents. The design includes a freeze calendar and CI linting to enforce the contract, aiming to reduce human error under pressure.", "body_md": "The first five commands after a page should be sealed, read-only, and boring, because creativity at 3 a.m. is how incidents get worse. I want every alert to land on a contract, not a chat transcript that an agent can rewrite. Have you noticed how quickly an assistant jumps from checking latency to restarting a deployment under pressure? That jump is the real bug, and a linter can catch it before the next on-call rotation starts.\n\nAgentic tools are getting better at sounding like they belong in the incident channel. They also assume missing facts, which is the same failure people keep reporting when they wire agents to cloud APIs. Why should a paging bot invent `kubectl rollout restart`\n\nwhen your freeze calendar already forbids writes? Why should the first command after a page ever be a mutation?\n\nI treat the opening of a runbook as a sealed list, not as a prompt the model can keep editing. The model may draft the narrative, the diagrams, and the paragraph that explains why the alert exists. It does not get to author the first commands, the escalation timer, or the freeze switch. Those three fields are the contract, and they belong in YAML that a test can fail in CI.\n\nI keep four blocks, and I refuse to ship a runbook that is missing any of them. Does your current wiki page even list the first command as a string you can grep? If not, an assistant will happily fill the gap with something that sounds like expertise.\n\nThe order matters more than the prose. Humans under a page will run whatever sits at the top, and models will imitate that habit. If the top is sealed and dull, the rest of the incident has a chance to stay honest.\n\nThe following example is a proposal you can save as `runbooks/checkout-latency.yaml`\n\nand validate locally. It is not a production dump from a company, and I am not claiming it pages a real cluster.\n\n```\napiVersion: oncall.contract/v1\nid: checkout-latency-p1\nowner: sre-payments\nalert:\n  name: CheckoutLatencyHigh\n  severity: p1\n  page_if:\n    duration_minutes_gte: 5\n    error_budget_remaining_pct_lt: 25\n  ticket_if:\n    duration_minutes_lt: 5\nfirst_commands:\n  sealed: true\n  max_steps: 5\n  mode: read_only\n  steps:\n    - id: fc1\n      purpose: confirm the alert is still firing\n      cmd: kubectl -n checkout get deploy checkout-api -o jsonpath='{.status.conditions}'\n      timeout_sec: 20\n    - id: fc2\n      purpose: check replica readiness without touching pods\n      cmd: kubectl -n checkout get pods -l app=checkout-api -o wide\n      timeout_sec: 20\n    - id: fc3\n      purpose: read recent warning events only\n      cmd: kubectl -n checkout get events --field-selector type=Warning --sort-by=.lastTimestamp\n      timeout_sec: 30\n    - id: fc4\n      purpose: sample current latency without mutating anything\n      cmd: curl -fsS --max-time 5 http://127.0.0.1:9090/api/v1/query?query=up\n      timeout_sec: 10\n    - id: fc5\n      purpose: confirm freeze state before anyone talks about restarts\n      cmd: python3 tools/freeze_status.py --service checkout-api\n      timeout_sec: 10\nescalation:\n  after_minutes: 10\n  to_role: secondary-oncall\n  require_human: true\n  stop_if: alert_resolved\nfreeze:\n  calendar: ./calendars/checkout-freeze.yaml\n  during_freeze:\n    allow_mutating_commands: false\n    page_still_allowed: true\n  unfreeze:\n    requires_roles: [incident-commander, sre-lead]\n    min_approvers: 2\n    ttl_minutes: 30\n    note: Unfreeze unlocks the mutating allowlist only, never the first_commands seal.\n```\n\nLook at `first_commands.sealed`\n\n. That boolean is the whole point of the file. If a generator rewrites those steps, the linter should fail the pull request, not the incident. Would you merge a runbook that lets a chatbot swap `get`\n\nfor `delete`\n\nbecause the prompt said be helpful?\n\nPair the runbook with a calendar file. Empty rumors in Slack are not a freeze source of truth.\n\n```\n# calendars/checkout-freeze.yaml\nservice: checkout-api\ntimezone: UTC\nwindows:\n  - id: sep-freeze-2026\n    start: 2026-09-01T00:00:00Z\n    end: 2026-09-08T00:00:00Z\n    reason: payment cutover\nmutating_allowlist_when_unfrozen:\n  - kubectl -n checkout rollout restart deploy/checkout-api\n```\n\nI do not want freeze to mean please be careful tonight. I want freeze to mean the mutating allowlist is empty until two humans spend a TTL. Unfreeze is an explicit event with two roles and a short clock, because a permanent unfreeze is just another wiki lie that nobody re-reads.\n\n| State | First commands | Mutating remediations | Page? | Who can change the lock |\n|---|---|---|---|---|\n| freeze | sealed, read-only | blocked | yes | two roles, TTL |\n| unfrozen | still sealed, read-only | allowlist only | yes | lock returns after TTL |\n| no calendar | fail closed | fail closed | ticket only | owner must add a calendar |\n\nWould you let an agent flip that table because the system prompt said be helpful? I would not, and neither should your pager. During freeze, the only honest remediation text is collect evidence, then wait for a named unfreeze. After a valid unfreeze, you may run a separate allowlisted command, never a rewritten first command. The seal does not thaw when the lock opens. That is the rule I want sitting next to every chatbot integration.\n\nSave this as `tools/lint_runbook.py`\n\n. Treat it as a local check you can run in CI. I am not publishing pass or fail metrics from a fleet I do not have.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Lint an on-call runbook contract. Proposal: run against YAML in ./runbooks.\"\"\"\nfrom __future__ import annotations\n\nimport sys\nfrom pathlib import Path\n\ntry:\n    import yaml\nexcept ImportError:\n    print(\"Install pyyaml: pip install pyyaml\", file=sys.stderr)\n    sys.exit(2)\n\nMUTATING_TOKENS = (\n    \" delete \",\n    \" apply \",\n    \" patch \",\n    \" scale \",\n    \" rollout restart\",\n    \" drain \",\n    \" cordon \",\n    \" rm \",\n    \" drop \",\n)\n\nREAD_VERBS = (\" get \", \" describe \", \" logs \", \" top \")\n\ndef is_read_only(cmd: str) -> bool:\n    compact = \" \".join(cmd.split())\n    padded = f\" {compact.lower()} \"\n    if any(tok in padded for tok in MUTATING_TOKENS):\n        return False\n    if compact.startswith(\"kubectl\") and not any(v in f\" {compact} \" for v in READ_VERBS):\n        return False\n    return compact.startswith((\"kubectl\", \"curl -fsS\", \"curl -sS\", \"python3 tools/freeze_status.py\"))\n\ndef lint(doc: dict) -> list[str]:\n    errors: list[str] = []\n    alert = doc.get(\"alert\") or {}\n    if not alert.get(\"name\") or not alert.get(\"severity\"):\n        errors.append(\"alert.name and alert.severity are required\")\n    if not (alert.get(\"page_if\") or alert.get(\"ticket_if\")):\n        errors.append(\"alert must declare page_if or ticket_if\")\n\n    fc = doc.get(\"first_commands\") or {}\n    if fc.get(\"sealed\") is not True:\n        errors.append(\"first_commands.sealed must be true\")\n    if fc.get(\"mode\") != \"read_only\":\n        errors.append(\"first_commands.mode must be read_only\")\n    steps = fc.get(\"steps\") or []\n    max_steps = int(fc.get(\"max_steps\") or 5)\n    if not 1 <= len(steps) <= max_steps:\n        errors.append(\"first_commands.steps must contain 1..max_steps entries\")\n    for step in steps:\n        cmd = str(step.get(\"cmd\") or \"\")\n        if not is_read_only(cmd):\n            errors.append(f\"mutating or unknown first command: {cmd}\")\n        if int(step.get(\"timeout_sec\") or 0) <= 0:\n            errors.append(f\"step {step.get('id')} needs timeout_sec\")\n\n    esc = doc.get(\"escalation\") or {}\n    if int(esc.get(\"after_minutes\") or 0) <= 0:\n        errors.append(\"escalation.after_minutes must be > 0\")\n    if not esc.get(\"require_human\"):\n        errors.append(\"escalation.require_human must be true\")\n    if not esc.get(\"to_role\"):\n        errors.append(\"escalation.to_role is required\")\n\n    freeze = doc.get(\"freeze\") or {}\n    if not freeze.get(\"calendar\"):\n        errors.append(\"freeze.calendar is required (fail closed)\")\n    during = freeze.get(\"during_freeze\") or {}\n    if during.get(\"allow_mutating_commands\") is not False:\n        errors.append(\"during_freeze.allow_mutating_commands must be false\")\n    unfreeze = freeze.get(\"unfreeze\") or {}\n    if int(unfreeze.get(\"min_approvers\") or 0) < 2:\n        errors.append(\"unfreeze.min_approvers must be >= 2\")\n    if int(unfreeze.get(\"ttl_minutes\") or 0) <= 0:\n        errors.append(\"unfreeze.ttl_minutes must be > 0\")\n    note = str(unfreeze.get(\"note\") or \"\").lower()\n    if \"never\" not in note or \"first_commands\" not in note:\n        errors.append(\"unfreeze note must say the first_commands seal never thaws\")\n    return errors\n\ndef main(argv: list[str]) -> int:\n    if len(argv) != 2:\n        print(\"usage: lint_runbook.py <runbook.yaml>\", file=sys.stderr)\n        return 2\n    path = Path(argv[1])\n    doc = yaml.safe_load(path.read_text())\n    errors = lint(doc)\n    if errors:\n        print(f\"FAIL {path}\")\n        for item in errors:\n            print(f\"  - {item}\")\n        return 1\n    print(f\"PASS {path}\")\n    return 0\n\nif __name__ == \"__main__\":\n    raise SystemExit(main(sys.argv))\n```\n\nA matching stub keeps `fc5`\n\nreproducible without pretending you already have a freeze service.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Proposal stub: print freeze state from the calendar file.\"\"\"\nimport argparse\nfrom datetime import datetime, timezone\nfrom pathlib import Path\n\nimport yaml\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"--service\", required=True)\nparser.add_argument(\"--calendar\", default=\"calendars/checkout-freeze.yaml\")\nargs = parser.parse_args()\n\ndoc = yaml.safe_load(Path(args.calendar).read_text())\nnow = datetime.now(timezone.utc)\nactive = False\nfor window in doc.get(\"windows\") or []:\n    start = datetime.fromisoformat(window[\"start\"].replace(\"Z\", \"+00:00\"))\n    end = datetime.fromisoformat(window[\"end\"].replace(\"Z\", \"+00:00\"))\n    if start <= now < end and doc.get(\"service\") == args.service:\n        active = True\n        print(f\"FREEZE active id={window['id']} reason={window['reason']}\")\nif not active:\n    print(\"FREEZE inactive\")\n```\n\nRun the happy path, then break it on purpose:\n\n```\npip install pyyaml\npython3 tools/lint_runbook.py runbooks/checkout-latency.yaml\n# expected: PASS runbooks/checkout-latency.yaml\n```\n\nCopy the YAML, change `fc1`\n\nto `kubectl -n checkout rollout restart deploy/checkout-api`\n\n, and run the linter again. If it does not exit 1, you do not have a seal. You have a suggestion that will fire during a real page.\n\nI still want help writing the why this alert exists section, because humans abandon runbooks that read like legal documents. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is enough for the drafting half of this workflow if you do not want that loop on a production laptop.\n\nThe prompt I actually send is intentionally rude to the model, because polite prompts invite invented flags.\n\n```\nYou may rewrite purpose strings and the human-readable summary.\nYou may not add, remove, or edit first_commands.steps[].cmd.\nYou may not set sealed to false.\nYou may not lower unfreeze.min_approvers.\nYou may not invent kubectl flags that are not already in the file.\nReturn YAML only if lint_runbook.py would still pass.\n```\n\nIf the model cannot obey that, I paste the prose back by hand and leave the commands untouched. Have you tried asking an agent to stop being helpful? It is a surprisingly good filter for on-call text.\n\nA tiny workflow that stays honest looks like this:\n\n`lint_runbook.py`\n\nruns in CI on every pull request that touches `runbooks/`\n\n.That is the whole loop. No dashboard screenshots, no invented latency wins, and no claim that the bot closed the incident.\n\nSkip this contract if you do not own the cluster, because a sealed `kubectl`\n\nlist is still a credential problem. Skip it if your alerts are not mapped to a single service owner, because escalation to a role that does not exist is theater. Skip it if you need the bot to execute remediations unattended, since this design refuses that job on purpose.\n\nAlso skip it if your freeze calendar is a rumor in chat. The linter fail-closes without a calendar path, and that will annoy you until you write the file. Good. Annoyance before the page is cheaper than a restart during a payment cutover.\n\nThis linter does not prove a command is safe. It only proves the string looks read-only and the freeze block is present. A `curl`\n\nto the wrong internal URL can still be harmful, and a `kubectl logs`\n\non a huge pod can still stall a laptop. Timeouts are a courtesy, not a sandbox, and they will not save you from a bad kubeconfig.\n\nI also cannot claim a specific model name, quota, or hardware profile for the drafting step, because those numbers go stale and I will not invent them. The value is the seal, not the vendor sticker. If you strip every product name out of this article, you should still have a YAML file, a calendar, and a Python exit code.\n\nOne more uncomfortable question sits under all of this. If your runbook cannot fail a unit test, why would you let an agent touch it during a page?", "url": "https://wpnews.pro/news/seal-the-first-five-commands-before-your-on-call-bot-invents-a-restart", "canonical_source": "https://dev.to/appcpp_9071/seal-the-first-five-commands-before-your-on-call-bot-invents-a-restart-328m", "published_at": "2026-09-04 06:37:15+00:00", "updated_at": "2026-09-04 06:53:43.068095+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/seal-the-first-five-commands-before-your-on-call-bot-invents-a-restart", "markdown": "https://wpnews.pro/news/seal-the-first-five-commands-before-your-on-call-bot-invents-a-restart.md", "text": "https://wpnews.pro/news/seal-the-first-five-commands-before-your-on-call-bot-invents-a-restart.txt", "jsonld": "https://wpnews.pro/news/seal-the-first-five-commands-before-your-on-call-bot-invents-a-restart.jsonld"}}