If your AI agent owns tools with side effects, one question decides whether it
is safe to ship: what happens when the model confidently calls a money tool on
invented grounds.
This is a writeup of one mechanism that closes that hole, and of where the
mechanism stops working. The context is assistants that talk to real customers
in messengers and can do irreversible things: confirm a payment, issue an
invoice, book a slot, notify the business owner.
A line like "only confirm payment after you received the receipt" executes
with a probability, not with a guarantee. That is not a quality problem with
the model. It follows from the training objective: be helpful, agree with the
person in front of you.
The conversation goes like this. The customer writes "I already paid, I will
send the receipt later, please confirm". There is no receipt. The model sees a
polite persistent human, sees an instruction that contradicts him, and over a
long context it picks cooperation. It answers "payment confirmed" and calls
the tool.
For tone of voice, probabilistic execution is fine. For money it is not.
One more hope worth killing early: the tool config field that looks like a
predicate. Most function schemas carry something like trigger_type
, and
ai_decides
literally means "the model decides". That field controls when the
tool is offered, never under which facts the tool is allowed to fire.
The idea is small. A function carries a list of facts that the executor checks
against the database before dispatch. Not "the model believes a receipt
exists", but "there is an inbound attachment in this conversation".
Stored as JSONB next to the function:
[
{ "type": "client_sent_media", "within_messages": 10, "media_kinds": ["image", "document"] },
{ "type": "lead_field_filled", "field": "phone" }
]
The type list is deliberately short and covers nearly every real requirement:
type FunctionPrecondition =
| { type: "client_sent_media"; within_messages?: number; media_kinds?: string[] }
| { type: "lead_field_filled"; field: string }
| { type: "function_called_before"; name: string }
| { type: "min_client_messages"; count: number };
client_sent_media
scans the last N messages written by the customer rather
than the last N rows of the thread. An owner who configures "the last 10
messages" means ten customer replies, not ten rows half of which the bot wrote
itself. The window is capped by a constant so that within_messages: 100000
in a config cannot turn the check into a table scan.
function_called_before
reads the event log and requires an earlier
successful call in the same conversation. That is how you build chains like
"verify identity first, then modify the booking".
The check lives in exactly one place. It sits in the tool executor, after
argument validation and strictly before dispatch to any handler. Put it inside
the handlers instead and you fix the class one handler at a time, which means
the next money-touching tool ships without a guard.
A block is returned to the model as a tool error with a reason. Not a
silent refusal:
Blocked: the customer must have sent a image/document attachment in their
last 10 messages. This did NOT happen. Do not tell the customer it did.
Ask the customer for what is missing, then call this function again.
The difference matters more than it looks. After a silent refusal the model
assumes the call went through and keeps lying to the customer. An error with a
cause produces self correction inside the same round: the bot goes and asks
for the receipt.
The requirement is appended to the tool description, so the model sees it
before spending a call:
HARD REQUIREMENT: this function is blocked and will refuse to run unless the
customer must have sent a image/document attachment in their last 10 messages.
Do not claim the action happened until the call actually succeeds.
When the check itself throws, the call goes through. It is not blocked.
} catch (err) {
logger.error("Precondition check failed, letting the call through", { ... });
}
Here is the reasoning. A precondition defends against model hallucination, not
against an attacker. An attacker has no reach into this layer at all: he
speaks to the bot in words, while the facts come from our own database. So the
failure mode should be chosen by cost. Blocking every function for every
customer because Postgres blinked means breaking live conversations (no
invoice, no booking, no answer) over a hypothesis. The failure goes loudly
into the log, and the decision falls back to the prompt, exactly as it was
before the guard existed.
If this were access control the choice would be the opposite, fail closed. It
is not access control, and pretending otherwise would be worse than having no
guard.
It does not replace authorization, idempotency or rate limits. It answers one
question: is there a fact in this conversation without which the action makes
no sense.
It does not rescue a badly specified function. If your only guard is
min_client_messages: 2
, you moved the problem one message down the road.
It costs nothing where it is not used. A function with an empty precondition
list issues zero queries, the branch returns on an empty array. That property
is what keeps the guard alive past the second release: a check that slows down
every conversation for the sake of one money flow gets removed by whoever is
on call.
An LLM in production behaves like a capable intern. Most of its calls are
good, and nobody lets an intern sign the cheques. Boundaries belong in code,
get verified against data, and get logged in a way that survives a restart.
The prompt owns the quality of the conversation, and nothing beyond it.
All of the above runs in the platform I build, DOS AI: AI assistants for
WhatsApp and Telegram with a built in CRM, configured in plain text. If you
are building your own, our REST API, webhooks and MCP server are public, so
you can plug your agent in and look at the contract from the outside. The
machine readable spec sits at https://dosai.pro/llms.txt and the code samples
are on GitHub: https://github.com/adsytd1/dosai-api
Happy to go deeper on failure modes in the comments.