{"slug": "how-we-built-an-autonomous-incident-war-room-with-switch-ai", "title": "How We Built an Autonomous Incident War Room with Switch AI", "summary": "Developer Krish Punia built OpsSwitch, an open-source multi-agent incident-response system on the Switch AI platform that runs a four-agent squad inside team chat to triage, investigate, patch, and document production incidents. The agents—sentinel, sleuth, releaser, and scribe—are defined in a YAML template and communicate over SSE and HTTP, with a human-in-the-loop gate requiring explicit approval before any production deploy. In a demo against a cart-service KeyError bug, the relay moved from alert to root-cause commit identification and a proposed hotfix, with a post-mortem drafted after human sign-off.", "body_md": "How We Built an Autonomous Incident War Room with Switch AI\n\nPublished for the WeMakeDevs × DevTools Academy \"Flip the Switch\" Hackathon\n\nBy Krish Punia\n\nImagine getting paged at 3:15 AM on a Saturday. Production is throwing HTTP 500s on the checkout endpoint. You log onto Slack, ping three teammates, search Datadog for error traces, sift through dozens of recent GitHub commits, and debate whether to rollback or hotfix.\n\nBy the time the patch is live and the post-mortem is written, over 45 minutes have elapsed.\n\nWhat if your team chat wasn't just a place where humans panic, but an active War Room where autonomous AI agents collaborate alongside you to diagnose, reproduce, patch, and document the incident in under 90 seconds?\n\nIn this article, I'll walk you through how I built OpsSwitch using Switch AI—an open-source platform that brings AI coding agents directly into team chat.\n\nMost AI coding tools (Claude Code, Cursor, Codex, OpenCode) live isolated inside a developer's terminal or IDE. But real engineering happens in teams, and team coordination happens in chat (Slack, Mattermost, Microsoft Teams, Discord).\n\nSwitch solves this by providing:\n\nChat as a Room: Every channel becomes an environment where multiple agents and humans coexist.\n\nFirst-Class Agent Identity: Agents get their own avatars, names, and addressing rules (addressing: anyone or addressing: owner-agents).\n\nAgent Protocol: Agents communicate via Server-Sent Events (SSE) and HTTP, allowing any custom agent or LLM framework to participate.\n\nHuman-in-the-Loop Governance: Built-in gates ensure that dangerous actions (like production deploys) require explicit human sign-off.\n\nIn OpsSwitch, we created a specialized four-agent squad:\n\n[ PagerDuty Alert ]\n\n        │\n\n        ▼\n\n[@ops-sentinel](https://dev.toCommander)  ──► Assesses SEV, summarizes impact\n\n        │\n\n        ▼\n\n[@ops-sleuth](https://dev.toInvestigator)──► Git blame, commit diff, root cause\n\n        │\n\n        ▼\n\n[@ops-releaser](https://dev.toRemediation) ──► Generates patch, verifies tests\n\n        │\n\n        ▼\n\n [Human Engineer](https://dev.toGate)      ──► Replies \"Approve\"\n\n        │\n\n        ▼\n\n[@ops-scribe](https://dev.toChronicler)  ──► Markdown Post-Mortem delivered\n\nThe Switch Template (template.yaml)\n\nSwitch allows you to define your entire multi-agent topology in a clean YAML template:\n\nversion: 1\n\nparams:\n\n  team:\n\n    type: string\n\n    default: ops\n\n  repo_path:\n\n    type: string\n\n    default: \"{$agents_dir}/demo-service\"\n\nagents:\n\nname: \"{team}-sentinel\"\n\ndescription: Incident Commander\n\naddressing: anyone\n\ninstructions: |\n\n  Triage incoming error reports. Assign SEV-1, SEV-2, or SEV-3.\n\n  Hand off code investigation to @{team}-sleuth with the exact stack trace.\n\nname: \"{team}-sleuth\"\n\ndescription: Git & Code Investigator\n\naddressing: owner-agents\n\ninstructions: |\n\n  Inspect the repository. Run git log -n 5 and git blame.\n\n  Pinpoint the exact commit and line causing the failure.\n\n  Hand off to @{team}-releaser.\n\nname: \"{team}-releaser\"\n\ndescription: Remediation & Deployment\n\naddressing: owner-agents\n\ninstructions: |\n\n  Prepare the hotfix and run local tests.\n\n  SAFETY: Never deploy without explicit \"Approve\" from {$creator}.\n\nname: \"{team}-scribe\"\n\ndescription: Incident Chronicler\n\naddressing: owner-agents\n\ninstructions: |\n\n  Compile the comprehensive post-mortem report once remediation is confirmed.\n\nroom:\n\n  name: \"{team}-incident-war-room\"\n\n  agents: [\"{team}-sentinel\", \"{team}-sleuth\", \"{team}-releaser\", \"{team}-scribe\"]\n\n  users: [\"{$creator}\"]\n\nThe Incident\n\nIn our test microservice (cart-service), a junior engineer pushed commit 8bae34d:\n\ndiscount = COUPONS[cart.coupon_code]  # Raises KeyError when coupon is invalid!\n\nThe Autonomous Relay in Action\n\nDetection: An error alert hits #incident-war-room:\n\nCRITICAL: 500 Spike on /api/v2/checkout (KeyError: 'INVALID_PROMO')\n\nTriage: @ops-sentinel declares SEV-2, notes that unauthenticated shoppers with promo codes are blocked, and addresses @ops-sleuth.\n\nInvestigation: @ops-sleuth runs git blame and catches commit 8bae34d within 15 seconds. It notes that COUPONS.get(cart.coupon_code, 0.0) was needed.\n\nRemediation & Human Gate: @ops-releaser prepares the patch, executes python3 test_service.py to confirm 2/2 tests pass, and tags me:\n\n\"[@krish](https://dev.to/krish) Punia: Hotfix verified locally. Reply 'Approve' to deploy.\"\n\nApproval & Deploy: I review the one-line diff and type \"Approve\".\n\nDocumentation: @ops-scribe posts a complete Markdown post-mortem with 5 Whys, timeline, and action items.\n\nTotal resolution time: 91 seconds.\n\nMulti-Agent Specialization Beats Monolithic Prompts: Giving each agent a distinct role (Sentinel vs. Sleuth vs. Releaser vs. Scribe) prevented context pollution and hallucinations.\n\nHuman Gates Are Mandatory for Enterprise Trust: Autonomous agents must be safe by design. Switch's addressing model makes human sign-offs natural and reliable.\n\nChat is the Natural Home for AI Teammates: Instead of switching between 5 browser tabs and terminals during an emergency, having your agents live in your team chat keeps everyone aligned.\n\nYou can explore the complete repository, Switch template, and reproduction harness here: 👉 GitHub: kpunia306/ops-switch\n\nSpecial thanks to WeMakeDevs, DevTools Academy, and the Switch AI team for hosting this fantastic hackathon!", "url": "https://wpnews.pro/news/how-we-built-an-autonomous-incident-war-room-with-switch-ai", "canonical_source": "https://dev.to/krish_punia_0205fafa9acf1/how-we-built-an-autonomous-incident-war-room-with-switch-ai-5680", "published_at": "2026-11-06 15:21:23+00:00", "updated_at": "2026-11-06 15:24:57.016228+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "ai-products"], "entities": ["Krish Punia", "Switch AI", "OpsSwitch", "WeMakeDevs", "DevTools Academy", "Slack", "Datadog", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-we-built-an-autonomous-incident-war-room-with-switch-ai", "markdown": "https://wpnews.pro/news/how-we-built-an-autonomous-incident-war-room-with-switch-ai.md", "text": "https://wpnews.pro/news/how-we-built-an-autonomous-incident-war-room-with-switch-ai.txt", "jsonld": "https://wpnews.pro/news/how-we-built-an-autonomous-incident-war-room-with-switch-ai.jsonld"}}