{"slug": "your-prompt-is-not-a-security-boundary", "title": "Your prompt is not a security boundary", "summary": "A developer proposes a mechanism to prevent AI agents from executing irreversible actions, such as confirming payments, based on unverified claims. The approach involves attaching preconditions to tool functions that are checked against the database before dispatch, ensuring actions only occur when factual conditions are met. The mechanism is implemented in the tool executor and returns a block error to the model, prompting self-correction.", "body_md": "If your AI agent owns tools with side effects, one question decides whether it\n\nis safe to ship: what happens when the model confidently calls a money tool on\n\ninvented grounds.\n\nThis is a writeup of one mechanism that closes that hole, and of where the\n\nmechanism stops working. The context is assistants that talk to real customers\n\nin messengers and can do irreversible things: confirm a payment, issue an\n\ninvoice, book a slot, notify the business owner.\n\nA line like \"only confirm payment after you received the receipt\" executes\n\nwith a probability, not with a guarantee. That is not a quality problem with\n\nthe model. It follows from the training objective: be helpful, agree with the\n\nperson in front of you.\n\nThe conversation goes like this. The customer writes \"I already paid, I will\n\nsend the receipt later, please confirm\". There is no receipt. The model sees a\n\npolite persistent human, sees an instruction that contradicts him, and over a\n\nlong context it picks cooperation. It answers \"payment confirmed\" and calls\n\nthe tool.\n\nFor tone of voice, probabilistic execution is fine. For money it is not.\n\nOne more hope worth killing early: the tool config field that looks like a\n\npredicate. Most function schemas carry something like `trigger_type`\n\n, and\n\n`ai_decides`\n\nliterally means \"the model decides\". That field controls when the\n\ntool is offered, never under which facts the tool is allowed to fire.\n\nThe idea is small. A function carries a list of facts that the executor checks\n\nagainst the database before dispatch. Not \"the model believes a receipt\n\nexists\", but \"there is an inbound attachment in this conversation\".\n\nStored as JSONB next to the function:\n\n```\n[\n  { \"type\": \"client_sent_media\", \"within_messages\": 10, \"media_kinds\": [\"image\", \"document\"] },\n  { \"type\": \"lead_field_filled\", \"field\": \"phone\" }\n]\n```\n\nThe type list is deliberately short and covers nearly every real requirement:\n\n```\ntype FunctionPrecondition =\n  | { type: \"client_sent_media\"; within_messages?: number; media_kinds?: string[] }\n  | { type: \"lead_field_filled\"; field: string }\n  | { type: \"function_called_before\"; name: string }\n  | { type: \"min_client_messages\"; count: number };\n```\n\n`client_sent_media`\n\nscans the last N messages written by the customer rather\n\nthan the last N rows of the thread. An owner who configures \"the last 10\n\nmessages\" means ten customer replies, not ten rows half of which the bot wrote\n\nitself. The window is capped by a constant so that `within_messages: 100000`\n\nin a config cannot turn the check into a table scan.\n\n`function_called_before`\n\nreads the event log and requires an earlier\n\nsuccessful call in the same conversation. That is how you build chains like\n\n\"verify identity first, then modify the booking\".\n\n**The check lives in exactly one place.** It sits in the tool executor, after\n\nargument validation and strictly before dispatch to any handler. Put it inside\n\nthe handlers instead and you fix the class one handler at a time, which means\n\nthe next money-touching tool ships without a guard.\n\n**A block is returned to the model as a tool error with a reason.** Not a\n\nsilent refusal:\n\n```\nBlocked: the customer must have sent a image/document attachment in their\nlast 10 messages. This did NOT happen. Do not tell the customer it did.\nAsk the customer for what is missing, then call this function again.\n```\n\nThe difference matters more than it looks. After a silent refusal the model\n\nassumes the call went through and keeps lying to the customer. An error with a\n\ncause produces self correction inside the same round: the bot goes and asks\n\nfor the receipt.\n\n**The requirement is appended to the tool description**, so the model sees it\n\nbefore spending a call:\n\n```\nHARD REQUIREMENT: this function is blocked and will refuse to run unless the\ncustomer must have sent a image/document attachment in their last 10 messages.\nDo not claim the action happened until the call actually succeeds.\n```\n\nWhen the check itself throws, the call goes through. It is not blocked.\n\n```\n} catch (err) {\n  logger.error(\"Precondition check failed, letting the call through\", { ... });\n}\n```\n\nHere is the reasoning. A precondition defends against model hallucination, not\n\nagainst an attacker. An attacker has no reach into this layer at all: he\n\nspeaks to the bot in words, while the facts come from our own database. So the\n\nfailure mode should be chosen by cost. Blocking every function for every\n\ncustomer because Postgres blinked means breaking live conversations (no\n\ninvoice, no booking, no answer) over a hypothesis. The failure goes loudly\n\ninto the log, and the decision falls back to the prompt, exactly as it was\n\nbefore the guard existed.\n\nIf this were access control the choice would be the opposite, fail closed. It\n\nis not access control, and pretending otherwise would be worse than having no\n\nguard.\n\nIt does not replace authorization, idempotency or rate limits. It answers one\n\nquestion: is there a fact in this conversation without which the action makes\n\nno sense.\n\nIt does not rescue a badly specified function. If your only guard is\n\n`min_client_messages: 2`\n\n, you moved the problem one message down the road.\n\nIt costs nothing where it is not used. A function with an empty precondition\n\nlist issues zero queries, the branch returns on an empty array. That property\n\nis what keeps the guard alive past the second release: a check that slows down\n\nevery conversation for the sake of one money flow gets removed by whoever is\n\non call.\n\nAn LLM in production behaves like a capable intern. Most of its calls are\n\ngood, and nobody lets an intern sign the cheques. Boundaries belong in code,\n\nget verified against data, and get logged in a way that survives a restart.\n\nThe prompt owns the quality of the conversation, and nothing beyond it.\n\nAll of the above runs in the platform I build, DOS AI: AI assistants for\n\nWhatsApp and Telegram with a built in CRM, configured in plain text. If you\n\nare building your own, our REST API, webhooks and MCP server are public, so\n\nyou can plug your agent in and look at the contract from the outside. The\n\nmachine readable spec sits at [https://dosai.pro/llms.txt](https://dosai.pro/llms.txt) and the code samples\n\nare on GitHub: [https://github.com/adsytd1/dosai-api](https://github.com/adsytd1/dosai-api)\n\nHappy to go deeper on failure modes in the comments.", "url": "https://wpnews.pro/news/your-prompt-is-not-a-security-boundary", "canonical_source": "https://dev.to/dosai/your-prompt-is-not-a-security-boundary-382e", "published_at": "2026-08-16 03:56:19+00:00", "updated_at": "2026-08-16 04:41:19.812515+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-products"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/your-prompt-is-not-a-security-boundary", "markdown": "https://wpnews.pro/news/your-prompt-is-not-a-security-boundary.md", "text": "https://wpnews.pro/news/your-prompt-is-not-a-security-boundary.txt", "jsonld": "https://wpnews.pro/news/your-prompt-is-not-a-security-boundary.jsonld"}}