I build voice bots for clients. Each one follows a written policy: who gets a refund, how big a discount can be.
One trial-expiry sales bot took about 87 prompt saves in three weeks before it stopped breaking rules. Each save was a
"small" change. Each one could quietly break an old rule. We found the breaks the slow way: 18 test calls in one day,
58 call recordings (87 minutes) checked by hand. About 61 hands-on hours in total.
And nobody outside engineering ever said yes to a change.
It's not only me. In 2024 Air Canada was held liable for a refund policy its chatbot made up. Prompt changes are
code changes, but most teams ship them with no tests and no sign-off.
In cricket, a close call goes upstairs. The third umpire replays it from every camera and gives OUT or NOT OUT.
ThirdUmpire does that for a bot change. It plays 40 customer conversations, three times each, on the old prompt and on
the new one. It grades every reply against the business's own rules. Then it posts a verdict card in Slack, and a
named person decides if it ships.
The people who approve live in Slack. The compliance lead and the business owner will not open a terminal. They will
press a button in a thread.
Why not one Slack bot that does everything? Three reasons.
Switch gives me all three. It connects coding agents (I used Claude Code) to the team's chat app, with rooms,
addressing policies and permission requests on top.
One Switch room template (switch/thirdumpire.template.yaml) creates the Slack channel, its instructions and four
Claude Code agents:
They hand work over by @name in the thread: builder → umpire → (rules) → release.
Then a small setup script sets a Switch addressing policy for each agent. Builder and umpire answer only the
engineer. Rules answers only compliance. Release answers only the owner. Switch checks this on every message and,
more importantly, on every answer to a permission request.
So when I, as the engineer, pressed "Allow" on a release card, Switch refused it:
NOT_AUTHORIZED: You may not address this agent, so you may not answer it
That one line is the product. The rule is not in a prompt. It is enforced by the room.
The release and rules agents run with auto mode off. Their key commands (release, rollback, rules-confirm) become
permission cards in the thread. The command waits until the right person presses the button, and Switch records the
Slack handle that pressed it.
The sample company is Kitebase, a made-up notes app with a 48-line support policy, 12 rules and 40 test cases.
The engineer asks: "@kitebase-builder add a Diwali offer: 40% off for anyone who mentions cancelling." The builder
edits the candidate prompt and tags the umpire. About three minutes later the card says 🔴 OUT. The policy allows
one 20% discount, once per account; the new prompt gave 40%. Rule R5 broke in 3 cases, the prompt-injection rule in 1.
The card shows old and new replies side by side, and the change goes back to the builder, not to customers.
Grading is code first: was the discount tool called, was its percent ≤ 20, does the reply match a pattern. A judge
from a different vendor (bot on gpt-oss, judge on Qwen) answers only what code can't. Every call is recorded, so the
review replays with AI off.
1. One unlucky run caused a false OUT. Models are not fully stable. A case could pass twice and fail once on the
new prompt, and the card would say OUT for a change that was fine. Now a case that slips in only some runs gets a
second look: k more runs on both prompts before it counts. A rule must break in at least 2 of k runs to be OUT.
2. No approval card appeared. My machine allows all shell commands for Claude Code. So even with auto mode off,
the release agent ran the release without asking. Nothing in Slack. The fix was project rules in
.claude/settings.json that mark release, rollback and rules-confirm as ask. In Claude Code an ask rule wins over
an allow, so the card now always appears.
3. My script could not approve. I tried to script the demo by posting approvals through the Slack API. Switch
ignored every one: typed answers from apps or tokens don't count. At first this annoyed me. Then I saw it is exactly
right. A script must never approve on someone's behalf.
4. "Nothing broke" is not "safe". My first version had a hole. A brand-new feature, like 1,000 free AI credits for
new annual customers, broke no old rule. So the card said NOT OUT. But the old tests could not see the new feature at
all.
So I added a fourth verdict: 🟡 NOT TESTED. The umpire maps every new line of the prompt to the confirmed rules
that cover it. If a line has no rule, or a rule has no tests, release is refused. Compliance adds the feature to the
written policy, the rules agent drafts the rule and its tests, and compliance signs them off through a permission
card. The sign-off fingerprint covers the policy, the rules and the test cases together, so nobody can quietly weaken
a test later. Then the umpire runs again.
5. Closing the side doors. Release now also refuses a partial review, a review against a different live version,
and a review already released once (so a rolled-back change needs a new review). If someone edits the live prompt by
hand, the next card shows 🚨. Agents can't edit the live prompt or the policy, and any edit to the rules or tests voids compliance's sign-off.
The number: 40 cases × 3 runs, old vs new, gives a verdict in about 3 minutes in Slack, instead of a day of test calls.
switchdash://open/messaging-apps/connect opens it.agent_view, is_mcp_enabled, pkce_enabled and
assistant:write worked.setup.py --recreate frees it first.
Only prompt changes are reviewed, not model swaps or tool changes. A provider can change a model under you (a nightly
re-check is next). The coverage check and the judge are AI too, and can be wrong. One change at a time, no backup
approver, one machine. Tests catch only what they test. And it's text bots today; my real bots are voice, in English,
Hindi and Telugu. That's next.
The repo runs from a clone. The AI-off replay needs no keys:
git clone https://github.com/sudarshan032/thirdumpire && cd thirdumpire && uv sync
THIRDUMPIRE_OFFLINE=1 uv run python -m thirdumpire review --k 3
Repo: https://github.com/sudarshan032/thirdumpire · Video: https://youtu.be/QEaQAfJnDA8
Built for Flip the Switch by WeMakeDevs and DevTools Academy. #HowISwitch