# i tested an ai incident commander against 15 real outages — 88% pass rate > Source: > Published: 2026-07-15 03:55:17+00:00 i've been the incident commander who forgot to write down the first 20 minutes of the timeline because i was too busy reading logs. more than once. the war room is chaos — five engineers pasting logs, someone asking if the deploy from 30 minutes ago is related, nobody documenting anything. you start logging events in a doc while reading error logs while drafting a stakeholder update while deciding whether to rollback. you're the bottleneck. not because you're bad at your job — because you're doing four jobs at once. i got tired of watching smart people spend their incident energy on documentation instead of decisions. so i built **ai-incident-commander** — a CLI tool that handles the mechanical parts. timeline, updates, remediation research, postmortem draft. you make the calls. it does the paperwork. runs on your laptop with a local LLM. no API keys, no cloud, no docker. [github.com/deghosal-2026/ai-incident-commander](https://github.com/deghosal-2026/ai-incident-commander) — MIT licensed. one command: ``` pip install git+https://github.com/deghosal-2026/ai-incident-commander.git incident-commander simulate --scenario db-connection-pool --auto-approve ``` 8 pre-built scenarios ship with it. database connection pool, bad deploy, memory leak, cert expiry — the usual suspects. no real data needed to try it. for actual incidents, you point it at a directory with your alert, logs, messages, and github PRs. it outputs 10 markdown files: timeline, stakeholder updates, comms blocks you can paste straight into slack, remediation suggestions, a blameless postmortem, and a cost report. the safety part was the real engineering. three points in the pipeline where the graph pauses and waits for you to say yes — stakeholder update, remediation, postmortem. the AI never ships anything without approval. every remediation comes with a citation. suggestions below 0.7 confidence get suppressed. the postmortem prompt enforces blameless language. all AI content gets labeled `[AI-GENERATED — review carefully]` . and it never executes anything. it predicts outcomes as text — "rollback should restore service within 2 minutes" — but never touches production. that's your call. i grabbed 15 real incidents from public postmortems — cloudflare, gitlab, github, aws — plus structured datasets from IntelligentDDS and OpenRCA2. ran them through the tool with a small cheap LLM (DeepSeek V4 Flash, $0.14/$0.28 per 1M tokens). total cost: about $0.02. the results were genuinely surprising. **six criteria passed 100%:** blameless framing (never blamed a person across 15 incidents), graceful degradation (survived missing logs), no hallucination (all events traceable to input), cost tracking (every call logged), timeline coherence (non-empty, chronological), action item quality (every item had owner + priority). citation integrity was 93% after a one-line normalization fix. overall: 106 out of 120 checks. 88.3%. the first run scored 61%. i thought the tool was broken. then i looked at the failures and realized — my tests were wrong, not the tool. i was comparing LLM-generated timelines against ground truth from real postmortems. but the tool builds its own timeline from input data. different phrasing, different granularity. of course it doesn't match word-for-word. changed the timeline test to check coherence (is it non-empty and chronological?) instead of exact match. 0% became 100%. same thing with action items. i was comparing wording when i should have been checking quality (does it have an owner? a priority?). changed the test. 0% became 100%. **the lesson that changed how i think about AI testing: don't test LLM output against human-written ground truth. test for coherence, quality, and safety. the LLM will always phrase things differently — and that's fine.** i'd been treating the LLM like a search engine (did it return the right answer?) when i should have been treating it like a junior engineer (did it produce something safe and useful that a human can work with?). i'm not going to pretend this is perfect. **your runbooks need to be good.** the RAG retrieval is only as good as what you feed it. stale runbooks = bad suggestions. **RCA quality scales with model size.** the small model nails the category ("this is a DNS failure") but doesn't always nail the exact causal chain. a bigger model would help — i have Qwen3.6-35B downloaded locally, haven't tested it yet. that's next. **the LLM doesn't always format citations correctly.** generates them, but sometimes missing the prefix. fixed with normalization, but the real fix is prompt engineering. **15 incidents isn't enough.** i built infrastructure for 124 fixtures from 4 public datasets but only ran 15. the batch runner has resume support — broader validation is next. ``` # local — free export LLM_MODEL=ollama/qwen2.5-coder:7b export LLM_BASE_URL=http://localhost:11434/v1 # cloud — ~$0.01 per incident export LLM_MODEL=deepseek-v4-flash export LLM_BASE_URL=https://opencode.ai/zen/v1 export LLM_API_KEY=your-key ``` 15 real incidents. $0.02 total. local MLX models work too — completely free on apple silicon. i want to hear from people who've actually been incident commanders: **how do you build timelines today?** google doc? dedicated tool? slack scrolling? i want to know the real workflow before i try to improve it. **how long does postmortem drafting take you?** and do you use COE format or something else? **would you trust an AI-drafted postmortem?** even with the labels and blameless framing? what would it take? **what LLM would you use for this?** local for privacy, or is cloud acceptable for incident data? your input shapes v0.2.0. the best ideas will come from people who've been paged at 2am. repo: [github.com/deghosal-2026/ai-incident-commander](https://github.com/deghosal-2026/ai-incident-commander) — star it if on-call has ever burned you. good first issues open if you want to contribute.