DRS for your AI: making a support bot wait for a person before it changes (Switch + Slack)

A developer built ThirdUmpire, a system that replays customer conversations against old and new bot prompts and grades replies against a business's own rules before posting a verdict card in Slack for a named approver to decide. The tool runs on Switch, which connects Claude Code agents to Slack with per-agent addressing policies that enforce who may approve a release, and it flagged a test discount change as a rule violation (40% off versus a 20% cap) before it reached customers.

I build voice bots for clients. Each one follows a written policy: who gets a refund, how big a discount can be. One trial-expiry sales bot took about 87 prompt saves in three weeks before it stopped breaking rules. Each save was a "small" change. Each one could quietly break an old rule. We found the breaks the slow way: 18 test calls in one day, 58 call recordings 87 minutes checked by hand. About 61 hands-on hours in total. And nobody outside engineering ever said yes to a change. It's not only me. In 2024 Air Canada was held liable for a refund policy its chatbot made up. Prompt changes are code changes, but most teams ship them with no tests and no sign-off. In cricket, a close call goes upstairs. The third umpire replays it from every camera and gives OUT or NOT OUT. ThirdUmpire does that for a bot change. It plays 40 customer conversations, three times each, on the old prompt and on the new one. It grades every reply against the business's own rules. Then it posts a verdict card in Slack, and a named person decides if it ships. The people who approve live in Slack. The compliance lead and the business owner will not open a terminal. They will press a button in a thread. Why not one Slack bot that does everything? Three reasons. Switch gives me all three. It connects coding agents I used Claude Code to the team's chat app, with rooms, addressing policies and permission requests on top. One Switch room template switch/thirdumpire.template.yaml creates the Slack channel, its instructions and four Claude Code agents: They hand work over by @name https://dev.to/name in the thread: builder → umpire → rules → release. Then a small setup script sets a Switch addressing policy for each agent. Builder and umpire answer only the engineer. Rules answers only compliance. Release answers only the owner. Switch checks this on every message and, more importantly, on every answer to a permission request. So when I, as the engineer, pressed "Allow" on a release card, Switch refused it: NOT AUTHORIZED: You may not address this agent, so you may not answer it That one line is the product. The rule is not in a prompt. It is enforced by the room. The release and rules agents run with auto mode off. Their key commands release, rollback, rules-confirm become permission cards in the thread. The command waits until the right person presses the button, and Switch records the Slack handle that pressed it. The sample company is Kitebase, a made-up notes app with a 48-line support policy, 12 rules and 40 test cases. The engineer asks: "@kitebase-builder add a Diwali offer: 40% off for anyone who mentions cancelling." The builder edits the candidate prompt and tags the umpire. About three minutes later the card says 🔴 OUT . The policy allows one 20% discount, once per account; the new prompt gave 40%. Rule R5 broke in 3 cases, the prompt-injection rule in 1. The card shows old and new replies side by side, and the change goes back to the builder, not to customers. Grading is code first: was the discount tool called, was its percent ≤ 20, does the reply match a pattern. A judge from a different vendor bot on gpt-oss, judge on Qwen answers only what code can't. Every call is recorded, so the review replays with AI off. 1. One unlucky run caused a false OUT. Models are not fully stable. A case could pass twice and fail once on the new prompt, and the card would say OUT for a change that was fine. Now a case that slips in only some runs gets a second look : k more runs on both prompts before it counts. A rule must break in at least 2 of k runs to be OUT. 2. No approval card appeared. My machine allows all shell commands for Claude Code. So even with auto mode off, the release agent ran the release without asking. Nothing in Slack. The fix was project rules in .claude/settings.json that mark release, rollback and rules-confirm as ask . In Claude Code an ask rule wins over an allow , so the card now always appears. 3. My script could not approve. I tried to script the demo by posting approvals through the Slack API. Switch ignored every one: typed answers from apps or tokens don't count. At first this annoyed me. Then I saw it is exactly right. A script must never approve on someone's behalf. 4. "Nothing broke" is not "safe". My first version had a hole. A brand-new feature, like 1,000 free AI credits for new annual customers, broke no old rule. So the card said NOT OUT. But the old tests could not see the new feature at all. So I added a fourth verdict: 🟡 NOT TESTED . The umpire maps every new line of the prompt to the confirmed rules that cover it. If a line has no rule, or a rule has no tests, release is refused. Compliance adds the feature to the written policy, the rules agent drafts the rule and its tests, and compliance signs them off through a permission card. The sign-off fingerprint covers the policy, the rules and the test cases together, so nobody can quietly weaken a test later. Then the umpire runs again. 5. Closing the side doors. Release now also refuses a partial review, a review against a different live version, and a review already released once so a rolled-back change needs a new review . If someone edits the live prompt by hand, the next card shows 🚨. Agents can't edit the live prompt or the policy, and any edit to the rules or tests voids compliance's sign-off. The number: 40 cases × 3 runs, old vs new, gives a verdict in about 3 minutes in Slack, instead of a day of test calls. switchdash://open/messaging-apps/connect opens it. agent view , is mcp enabled , pkce enabled and assistant:write worked. setup.py --recreate frees it first. Only prompt changes are reviewed, not model swaps or tool changes. A provider can change a model under you a nightly re-check is next . The coverage check and the judge are AI too, and can be wrong. One change at a time, no backup approver, one machine. Tests catch only what they test. And it's text bots today; my real bots are voice, in English, Hindi and Telugu. That's next. The repo runs from a clone. The AI-off replay needs no keys: git clone https://github.com/sudarshan032/thirdumpire && cd thirdumpire && uv sync THIRDUMPIRE OFFLINE=1 uv run python -m thirdumpire review --k 3 Repo: https://github.com/sudarshan032/thirdumpire https://github.com/sudarshan032/thirdumpire · Video: https://youtu.be/QEaQAfJnDA8 https://youtu.be/QEaQAfJnDA8 Built for Flip the Switch by WeMakeDevs and DevTools Academy. HowISwitch