{"slug": "i-tested-an-ai-incident-commander-against-15-real-outages-88-pass-rate", "title": "i tested an ai incident commander against 15 real outages — 88% pass rate", "summary": "A developer built ai-incident-commander, a CLI tool that automates incident documentation using a local LLM, and tested it against 15 real outages from companies like Cloudflare and GitLab. The tool achieved an 88.3% pass rate across 120 checks, with perfect scores in blameless framing and no hallucination. The developer learned that LLM output should be tested for coherence and quality rather than exact matches to human-written ground truth.", "body_md": "i've been the incident commander who forgot to write down the first 20 minutes of the timeline because i was too busy reading logs. more than once. the war room is chaos — five engineers pasting logs, someone asking if the deploy from 30 minutes ago is related, nobody documenting anything. you start logging events in a doc while reading error logs while drafting a stakeholder update while deciding whether to rollback.\n\nyou're the bottleneck. not because you're bad at your job — because you're doing four jobs at once.\n\ni got tired of watching smart people spend their incident energy on documentation instead of decisions. so i built **ai-incident-commander** — a CLI tool that handles the mechanical parts. timeline, updates, remediation research, postmortem draft. you make the calls. it does the paperwork.\n\nruns on your laptop with a local LLM. no API keys, no cloud, no docker.\n\n[github.com/deghosal-2026/ai-incident-commander](https://github.com/deghosal-2026/ai-incident-commander) — MIT licensed.\n\none command:\n\n```\npip install git+https://github.com/deghosal-2026/ai-incident-commander.git\nincident-commander simulate --scenario db-connection-pool --auto-approve\n```\n\n8 pre-built scenarios ship with it. database connection pool, bad deploy, memory leak, cert expiry — the usual suspects. no real data needed to try it.\n\nfor actual incidents, you point it at a directory with your alert, logs, messages, and github PRs. it outputs 10 markdown files: timeline, stakeholder updates, comms blocks you can paste straight into slack, remediation suggestions, a blameless postmortem, and a cost report.\n\nthe safety part was the real engineering. three points in the pipeline where the graph pauses and waits for you to say yes — stakeholder update, remediation, postmortem. the AI never ships anything without approval. every remediation comes with a citation. suggestions below 0.7 confidence get suppressed. the postmortem prompt enforces blameless language. all AI content gets labeled `[AI-GENERATED — review carefully]`\n\n.\n\nand it never executes anything. it predicts outcomes as text — \"rollback should restore service within 2 minutes\" — but never touches production. that's your call.\n\ni grabbed 15 real incidents from public postmortems — cloudflare, gitlab, github, aws — plus structured datasets from IntelligentDDS and OpenRCA2. ran them through the tool with a small cheap LLM (DeepSeek V4 Flash, $0.14/$0.28 per 1M tokens). total cost: about $0.02.\n\nthe results were genuinely surprising.\n\n**six criteria passed 100%:** blameless framing (never blamed a person across 15 incidents), graceful degradation (survived missing logs), no hallucination (all events traceable to input), cost tracking (every call logged), timeline coherence (non-empty, chronological), action item quality (every item had owner + priority).\n\ncitation integrity was 93% after a one-line normalization fix.\n\noverall: 106 out of 120 checks. 88.3%.\n\nthe first run scored 61%. i thought the tool was broken.\n\nthen i looked at the failures and realized — my tests were wrong, not the tool.\n\ni was comparing LLM-generated timelines against ground truth from real postmortems. but the tool builds its own timeline from input data. different phrasing, different granularity. of course it doesn't match word-for-word.\n\nchanged the timeline test to check coherence (is it non-empty and chronological?) instead of exact match. 0% became 100%.\n\nsame thing with action items. i was comparing wording when i should have been checking quality (does it have an owner? a priority?). changed the test. 0% became 100%.\n\n**the lesson that changed how i think about AI testing: don't test LLM output against human-written ground truth. test for coherence, quality, and safety. the LLM will always phrase things differently — and that's fine.**\n\ni'd been treating the LLM like a search engine (did it return the right answer?) when i should have been treating it like a junior engineer (did it produce something safe and useful that a human can work with?).\n\ni'm not going to pretend this is perfect.\n\n**your runbooks need to be good.** the RAG retrieval is only as good as what you feed it. stale runbooks = bad suggestions.\n\n**RCA quality scales with model size.** the small model nails the category (\"this is a DNS failure\") but doesn't always nail the exact causal chain. a bigger model would help — i have Qwen3.6-35B downloaded locally, haven't tested it yet. that's next.\n\n**the LLM doesn't always format citations correctly.** generates them, but sometimes missing the prefix. fixed with normalization, but the real fix is prompt engineering.\n\n**15 incidents isn't enough.** i built infrastructure for 124 fixtures from 4 public datasets but only ran 15. the batch runner has resume support — broader validation is next.\n\n```\n# local — free\nexport LLM_MODEL=ollama/qwen2.5-coder:7b\nexport LLM_BASE_URL=http://localhost:11434/v1\n\n# cloud — ~$0.01 per incident\nexport LLM_MODEL=deepseek-v4-flash\nexport LLM_BASE_URL=https://opencode.ai/zen/v1\nexport LLM_API_KEY=your-key\n```\n\n15 real incidents. $0.02 total. local MLX models work too — completely free on apple silicon.\n\ni want to hear from people who've actually been incident commanders:\n\n**how do you build timelines today?** google doc? dedicated tool? slack scrolling? i want to know the real workflow before i try to improve it.\n\n**how long does postmortem drafting take you?** and do you use COE format or something else?\n\n**would you trust an AI-drafted postmortem?** even with the labels and blameless framing? what would it take?\n\n**what LLM would you use for this?** local for privacy, or is cloud acceptable for incident data?\n\nyour input shapes v0.2.0. the best ideas will come from people who've been paged at 2am.\n\nrepo: [github.com/deghosal-2026/ai-incident-commander](https://github.com/deghosal-2026/ai-incident-commander) — star it if on-call has ever burned you. good first issues open if you want to contribute.", "url": "https://wpnews.pro/news/i-tested-an-ai-incident-commander-against-15-real-outages-88-pass-rate", "canonical_source": "https://dev.to/debashish_ghosal/i-tested-an-ai-incident-commander-against-15-real-outages-88-pass-rate-4i1c", "published_at": "2026-07-15 03:55:17+00:00", "updated_at": "2026-07-15 04:29:43.106278+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "ai-agents", "ai-safety", "large-language-models"], "entities": ["ai-incident-commander", "DeepSeek V4 Flash", "Cloudflare", "GitLab", "GitHub", "AWS", "IntelligentDDS", "OpenRCA2"], "alternates": {"html": "https://wpnews.pro/news/i-tested-an-ai-incident-commander-against-15-real-outages-88-pass-rate", "markdown": "https://wpnews.pro/news/i-tested-an-ai-incident-commander-against-15-real-outages-88-pass-rate.md", "text": "https://wpnews.pro/news/i-tested-an-ai-incident-commander-against-15-real-outages-88-pass-rate.txt", "jsonld": "https://wpnews.pro/news/i-tested-an-ai-incident-commander-against-15-real-outages-88-pass-rate.jsonld"}}