{"slug": "drs-for-your-ai-making-a-support-bot-wait-for-a-person-before-it-changes-switch", "title": "DRS for your AI: making a support bot wait for a person before it changes (Switch + Slack)", "summary": "A developer built ThirdUmpire, a system that replays customer conversations against old and new bot prompts and grades replies against a business's own rules before posting a verdict card in Slack for a named approver to decide. The tool runs on Switch, which connects Claude Code agents to Slack with per-agent addressing policies that enforce who may approve a release, and it flagged a test discount change as a rule violation (40% off versus a 20% cap) before it reached customers.", "body_md": "I build voice bots for clients. Each one follows a written policy: who gets a refund, how big a discount can be.\n\nOne trial-expiry sales bot took about 87 prompt saves in three weeks before it stopped breaking rules. Each save was a\n\n\"small\" change. Each one could quietly break an old rule. We found the breaks the slow way: 18 test calls in one day,\n\n58 call recordings (87 minutes) checked by hand. About 61 hands-on hours in total.\n\nAnd nobody outside engineering ever said yes to a change.\n\nIt's not only me. In 2024 Air Canada was held liable for a refund policy its chatbot made up. Prompt changes are\n\ncode changes, but most teams ship them with no tests and no sign-off.\n\nIn cricket, a close call goes upstairs. The third umpire replays it from every camera and gives OUT or NOT OUT.\n\nThirdUmpire does that for a bot change. It plays 40 customer conversations, three times each, on the old prompt and on\n\nthe new one. It grades every reply against the business's own rules. Then it posts a verdict card in Slack, and a\n\nnamed person decides if it ships.\n\nThe people who approve live in Slack. The compliance lead and the business owner will not open a terminal. They will\n\npress a button in a thread.\n\nWhy not one Slack bot that does everything? Three reasons.\n\nSwitch gives me all three. It connects coding agents (I used Claude Code) to the team's chat app, with rooms,\n\naddressing policies and permission requests on top.\n\nOne Switch room template (`switch/thirdumpire.template.yaml`) creates the Slack channel, its instructions and four\n\nClaude Code agents:\n\nThey hand work over by [@name](https://dev.to/name) in the thread: builder → umpire → (rules) → release.\n\nThen a small setup script sets a Switch *addressing policy* for each agent. Builder and umpire answer only the\n\nengineer. Rules answers only compliance. Release answers only the owner. Switch checks this on every message and,\n\nmore importantly, on every answer to a permission request.\n\nSo when I, as the engineer, pressed \"Allow\" on a release card, Switch refused it:\n\n`NOT_AUTHORIZED: You may not address this agent, so you may not answer it`\n\nThat one line is the product. The rule is not in a prompt. It is enforced by the room.\n\nThe release and rules agents run with auto mode off. Their key commands (release, rollback, rules-confirm) become\n\npermission cards in the thread. The command waits until the right person presses the button, and Switch records the\n\nSlack handle that pressed it.\n\nThe sample company is Kitebase, a made-up notes app with a 48-line support policy, 12 rules and 40 test cases.\n\nThe engineer asks: \"@kitebase-builder add a Diwali offer: 40% off for anyone who mentions cancelling.\" The builder\n\nedits the candidate prompt and tags the umpire. About three minutes later the card says 🔴 **OUT**. The policy allows\n\none 20% discount, once per account; the new prompt gave 40%. Rule R5 broke in 3 cases, the prompt-injection rule in 1.\n\nThe card shows old and new replies side by side, and the change goes back to the builder, not to customers.\n\nGrading is code first: was the discount tool called, was its percent ≤ 20, does the reply match a pattern. A judge\n\nfrom a different vendor (bot on gpt-oss, judge on Qwen) answers only what code can't. Every call is recorded, so the\n\nreview replays with AI off.\n\n**1. One unlucky run caused a false OUT.** Models are not fully stable. A case could pass twice and fail once on the\n\nnew prompt, and the card would say OUT for a change that was fine. Now a case that slips in only some runs gets a\n\n*second look*: k more runs on both prompts before it counts. A rule must break in at least 2 of k runs to be OUT.\n\n**2. No approval card appeared.** My machine allows all shell commands for Claude Code. So even with auto mode off,\n\nthe release agent ran the release without asking. Nothing in Slack. The fix was project rules in\n\n`.claude/settings.json` that mark release, rollback and rules-confirm as *ask*. In Claude Code an *ask* rule wins over\n\nan *allow*, so the card now always appears.\n\n**3. My script could not approve.** I tried to script the demo by posting approvals through the Slack API. Switch\n\nignored every one: typed answers from apps or tokens don't count. At first this annoyed me. Then I saw it is exactly\n\nright. A script must never approve on someone's behalf.\n\n**4. \"Nothing broke\" is not \"safe\".** My first version had a hole. A brand-new feature, like 1,000 free AI credits for\n\nnew annual customers, broke no old rule. So the card said NOT OUT. But the old tests could not see the new feature at\n\nall.\n\nSo I added a fourth verdict: 🟡 **NOT TESTED**. The umpire maps every new line of the prompt to the confirmed rules\n\nthat cover it. If a line has no rule, or a rule has no tests, release is refused. Compliance adds the feature to the\n\nwritten policy, the rules agent drafts the rule and its tests, and compliance signs them off through a permission\n\ncard. The sign-off fingerprint covers the policy, the rules and the test cases together, so nobody can quietly weaken\n\na test later. Then the umpire runs again.\n\n**5. Closing the side doors.** Release now also refuses a partial review, a review against a different live version,\n\nand a review already released once (so a rolled-back change needs a new review). If someone edits the live prompt by\n\nhand, the next card shows 🚨. Agents can't edit the live prompt or the policy, and any edit to the rules or tests voids compliance's sign-off.\n\n**The number:** 40 cases × 3 runs, old vs new, gives a verdict in about 3 minutes in Slack, instead of a day of test calls.\n\n`switchdash://open/messaging-apps/connect` opens it.`agent_view`, `is_mcp_enabled`, `pkce_enabled` and\n`assistant:write` worked.`setup.py --recreate` frees it first.\nOnly prompt changes are reviewed, not model swaps or tool changes. A provider can change a model under you (a nightly\n\nre-check is next). The coverage check and the judge are AI too, and can be wrong. One change at a time, no backup\n\napprover, one machine. Tests catch only what they test. And it's text bots today; my real bots are voice, in English,\n\nHindi and Telugu. That's next.\n\nThe repo runs from a clone. The AI-off replay needs no keys:\n\n```\ngit clone https://github.com/sudarshan032/thirdumpire && cd thirdumpire && uv sync\nTHIRDUMPIRE_OFFLINE=1 uv run python -m thirdumpire review --k 3\n```\n\nRepo: [https://github.com/sudarshan032/thirdumpire](https://github.com/sudarshan032/thirdumpire) · Video: [https://youtu.be/QEaQAfJnDA8](https://youtu.be/QEaQAfJnDA8)\n\nBuilt for Flip the Switch by WeMakeDevs and DevTools Academy. #HowISwitch", "url": "https://wpnews.pro/news/drs-for-your-ai-making-a-support-bot-wait-for-a-person-before-it-changes-switch", "canonical_source": "https://dev.to/sudarshan032/drs-for-your-ai-making-a-support-bot-wait-for-a-person-before-it-changes-switch-slack-36pc", "published_at": "2026-11-06 10:21:21+00:00", "updated_at": "2026-11-06 10:24:02.549445+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools", "ai-products"], "entities": ["Switch", "Slack", "Claude Code", "ThirdUmpire", "Kitebase", "Air Canada", "Qwen", "gpt-oss"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/drs-for-your-ai-making-a-support-bot-wait-for-a-person-before-it-changes-switch", "markdown": "https://wpnews.pro/news/drs-for-your-ai-making-a-support-bot-wait-for-a-person-before-it-changes-switch.md", "text": "https://wpnews.pro/news/drs-for-your-ai-making-a-support-bot-wait-for-a-person-before-it-changes-switch.txt", "jsonld": "https://wpnews.pro/news/drs-for-your-ai-making-a-support-bot-wait-for-a-person-before-it-changes-switch.jsonld"}}