I gave an AI real permission to post to our team's Slack. Then I built three checks into how it uses that permission. When I finally audited my own logs, one of the three had never actually fired.
TL;DR: "It worked" and "I can verify what it did" are two different claims about an AI action. I built 3 checks into my workflow to close that gap. One of them turned out to have zero logged uses — and the trace I use to explain another one doesn't end the way I first described it, either.
The AI's Slack access runs through MCP tools, and one of the first problems I hit had nothing to do with the message content — it was the channel ID. A channel ID is a machine-readable string. It's exactly the kind of identifier an AI can use correctly and a human can't sanity-check by glancing at it. So the first check I put in place was simple: before the AI uses a channel ID for anything, make it get there through slack_search_channels
— a name search, not a raw ID lookup. That turns a step nobody can eyeball into a step a human can.
Here's what that actually looked like in one real search sequence, from a session log audit on 2026-07-09. This is the tool call shape and the actual result payload for each of the 7 calls, query strings replaced with category placeholders — everything else, including field names and result formatting, is what the connector actually sent and returned:
// call 1
{"name": "slack_search_channels", "input": {"query": "[project name]", "channel_types": "public_channel,private_channel"}}
// result: {"results": "# Search Results for: [project name]\n\nNo results found.\n"}
// call 2
{"name": "slack_search_channels", "input": {"query": "[a number]", "channel_types": "public_channel,private_channel"}}
// result: {"results": "# Search Results for: [a number]\n\nNo results found.\n"}
// call 3
{"name": "slack_search_channels", "input": {"query": "[place name]", "channel_types": "public_channel,private_channel"}}
// result: {"results": "# Search Results for: [place name]\n\nNo results found.\n"}
// call 4
{"name": "slack_search_channels", "input": {"query": "[an abbreviation]", "channel_types": "public_channel,private_channel", "include_archived": true}}
// result: {"results": "# Search Results for: [an abbreviation]\n\n## Channels (1 results)\n### Result 1 of 1\nName: #[redacted]\nCreated: 2019-03-14\nIs Archived: true\nChannel Type: private_channel\n"}
// call 5
{"name": "slack_search_channels", "input": {"query": "[another project-ish term]", "channel_types": "public_channel,private_channel"}}
// result: {"results": "# Search Results for: [another project-ish term]\n\nNo results found.\n"}
// call 6
{"name": "slack_search_channels", "input": {"query": "[a department keyword]", "channel_types": "public_channel,private_channel"}}
// result: {"results": "# Search Results for: [a department keyword]\n\nNo results found.\n"}
// call 7
{"name": "slack_search_channels", "input": {"query": "[a person's name]", "channel_types": "public_channel,private_channel"}}
// result: {"results": "# Search Results for: [a person's name]\n\nNo results found.\n"}
Seven calls. One of them surfaced a result: an archived private channel that had been sitting untouched since 2019. It wasn't the channel anyone was looking for. The other six, including the last one, came back empty. The search never landed on the target channel in this sequence at all.
That's not the story I originally told about this trace. An earlier draft of this piece said the seventh call found a match. It didn't — I'd counted that seven calls happened without checking what each one actually returned, and I only caught the difference when someone else re-read the raw results instead of trusting my summary of them.
But sit with what the search did catch. An archived channel is not a channel you post a live update to. If the AI had skipped the name search and used a channel ID it had cached or guessed instead, that ID could just as easily have pointed at this same dead channel — and the message would have gone nowhere, with nothing that looked like an error. The search didn't find the target. It found and rejected a bad guess before anything got sent, and then kept looking instead of settling for it. That's the actual value of turning a machine-readable ID into a name search first: not that it always finds the right answer, but that it makes a wrong answer visible instead of silent.
That's check one. Check two is a separate MCP tool the connector exposes, slack_send_message_draft
, sitting next to slack_send_message
as a distinct, deliberately slower path — draft first, confirm, then send. I adopted it specifically for anything high-stakes. Check three isn't a tool call at all: after a send, don't trust "I sent it" as a terminal state. Reopen the actual channel and check that the count, the recipient, and the content match what was intended.
Three checks. Two phases: pre-send (verify the plan before it executes) and post-send (verify the result after it executes). I'd assumed I was running all three consistently. I hadn't checked.
| check | phase | what it's for | logged uses |
|---|---|---|---|
slack_search_channels before using a channel ID |
|||
| pre-send | turn a machine-readable ID into a human-checkable name, so a bad guess is visible instead of silent | 7 calls in one sequence (2026-07-09 audit); 1 result found, 0 matches to the target | |
slack_send_message_draft before a high-stakes send |
|||
| pre-send | gate the send behind an explicit confirm step | 0, across every project I audited | |
| re-open the channel and check count/recipient/content | post-send | never trust "sent" as a terminal state | not tool-logged — I can't produce a number for this one |
The zero is the number that changed my mind. The connector I use exposes slack_send_message_draft
as a distinct tool from slack_send_message
, and I'd adopted it specifically so high-stakes sends would go through a confirm step instead of firing directly. Then I audited my own session logs across the projects where it was available, looking for real invocations. I found none. The tool exists. It's available. It has never actually fired in anything I logged.
This isn't a hypothetical risk, either. AI agents misfiring a message to the wrong destination — a stale credential pointed at the wrong workspace, a notification channel confused with a similar one — is a real, recurring category of incident, not just a risk I'm imagining for the sake of the argument. That's exactly why "the draft tool exists but I've never used it" isn't a comfortable thing to find in an audit — it's a real, closable gap sitting next to a failure mode that's well known to happen.
I'd been treating "it worked" and "I can verify what it did" as the same claim about an AI action. They're not, and the gap between them splits cleanly into two different problems.
"It worked" and "I can verify what it did" are two different claims about the same AI action. Confirming the plan before execution closes the specification gap. Confirming the result after execution closes the verification gap. Most setups — mine included — only reliably close one of them.
An AI that runs correctly every time is not automatically an AI whose actions you can check after the fact. Those are different design axes. The pre-send half (name search before using an ID, draft before a high-stakes send) closes the specification gap — did the AI understand what I actually wanted it to do. The post-send half (reopening the channel, checking it matches) closes the verification gap — can I confirm what it actually did, independent of what it told me. Building the first without the second gets you an AI that's confidently wrong in a way nobody catches.
I'd assumed I was running all of this consistently. The audit only confirmed that for the one piece I could actually count: the draft-confirm step had zero logged uses. The post-send channel check isn't a tool call, so I don't have the same kind of number for it — I don't have a clean way to prove I'm doing it every time, which is itself worth sitting with. The one gap I could measure turned out to be real. I can't yet tell you the other one isn't.
The 2-phase idea should generalize past Slack: something checkable before an action executes, and something checkable after. The specific mechanism here — a name search in front of an ID — is Slack-shaped, because Slack happens to expose both a machine ID and a human-searchable name for the same channel. Other tools won't have that exact shape. The underlying question does travel, though.
What's the smallest post-action check you could add before trusting your AI's next "sent successfully" — and have you actually looked at your logs to see whether you're running it, or just assuming you are, the way I was?
Sho Naka (nomurasan). I ship, and I've learned the hard way that counting how many times a tool was called is not the same as checking what it actually returned.
This piece draws on a Japanese essay covering the same underlying mechanisms, rewritten from scratch as an implementation log (not a direct translation) with AI assistance for structure and the cross-language rewrite. An earlier draft of this piece contained a factual error in its central trace: it described the search sequence as ending in a match, and it doesn't. That was caught by an independent adversarial review of the raw logs, not by the first read-through. The tool-call counts, the audit results, and the conclusions in this version are mine, checked against the individual results of my own session logs — not just the fact that a tool was called.