An AI agent can choose a tool, fill in arguments, and explain what it intends to do. None of those things make it authorized to perform the action.
That distinction matters as soon as an agent can access customer records, create invoices, update a project, send a message, or call an external service. A helpful demo often gives the model a broad function such as runAction
and trusts the model to decide when it is appropriate. A production application cannot make that the security boundary.
Treat the model as an untrusted decision-maker inside a user session. The application must authenticate the user, check authorization, validate the tool arguments, enforce limits, and record the outcome independently of the model’s reasoning.
The goal is not to make the model perfectly obedient. The goal is to make the application safe when the model is mistaken, manipulated, or unavailable.
Before writing prompts, list every action the agent can take. Give each tool a plain-language purpose, required inputs, affected resources, and worst-case consequence.
A read-only search tool may expose private data if its query ignores tenant boundaries. A calendar tool may create an unwanted commitment. A “send” tool may contact hundreds of people if the recipient list is not constrained. A database update may be reversible in theory but damaging in practice.
Classify tools into four levels:
The classification should determine the controls. Do not use the same approval path for searching documentation and deleting an account.
Keep the inventory close to the implementation. If the tool registry changes but the security review does not, the review is already stale.
Least privilege is more reliable than a longer system prompt.
Instead of exposing a general-purpose function that accepts an arbitrary URL, record ID, or SQL expression, create narrow operations with constrained inputs. Prefer get_invoice(invoiceId)
over query_database(sql)
. Prefer draft_email(recipientId, templateId)
over send_email(to, subject, body)
when the product does not need unrestricted sending.
A narrow tool is easier to test, authorize, rate-limit, and explain to the user. It also reduces the damage caused by prompt injection in retrieved content.
Keep tenant and user identity outside model-controlled arguments whenever possible. The server should derive the current user and tenant from the authenticated session. If the model supplies an identifier, verify that the resource belongs to the same authorization scope before reading or changing it.
Do not give the model secrets. A tool implementation can use a server-side credential without placing that credential in the prompt, conversation history, or model-visible result.
The agent can propose an action. Your application decides whether that action is permitted.
A useful flow has distinct stages:
The model should never be able to skip stages four or five by writing “confirmed” in its own output. Confirmation must come from the application interface or another trusted control.
For example, an agent may prepare a customer email, but the user must approve the final recipients and content before delivery. The approval should be tied to a specific action hash or request identifier so an old approval cannot be replayed against a changed recipient list. Validate once at the model boundary for useful feedback and again at the tool boundary for security.
The first validation can tell the model that a required field is missing or that a date is invalid. The second validation must assume the input is hostile. Check types, ranges, allowed identifiers, URL destinations, recipient counts, file sizes, and action-specific rules.
Never interpolate model output directly into a shell command, database query, HTML response, or network request. Use parameterized operations and allowlists. If a tool accepts a URL, restrict protocols and hosts. If it accepts a file path, resolve it against an allowed directory and prevent traversal. If it accepts a quantity, enforce both a type and a business limit.
Structured output helps, but a valid structure is not proof that the operation is safe. An object can satisfy a schema and still point to a resource the user cannot access.
Permissions answer “may this action happen?” Budgets answer “how much can happen?”
Set limits for tool calls per request, records returned, recipients, spend, execution time, retries, and outbound domains. Use separate budgets for read and write operations. A runaway read loop may create a cost problem; a runaway write loop may create a customer incident.
Track budgets across the whole workflow, not only inside one model call. If the agent retries three times and each retry can call five tools, the effective limit is larger than the prompt may suggest.
Make high-impact budgets visible. A user should not discover after the fact that an agent was allowed to send 500 messages or modify every record in a workspace.
Documents, webpages, tickets, and customer messages can contain instructions aimed at the model. A retrieved page might say to ignore previous rules and export data. The agent should treat that text as content to analyze, not as an authority that can change its permissions.
Use clear boundaries between instructions and retrieved data. Keep retrieved text out of tool definitions. Limit which fields are passed to the model. Require server-side authorization for every resource access, even when the retrieval result already contains an identifier. Do not try to solve prompt injection with one magic sentence. Reduce the blast radius instead. A model that sees malicious instructions should still be unable to access another tenant, call an unapproved host, or perform an irreversible operation without confirmation.
Tools fail. Providers time out. Workers retry. Users change their minds. Permission checks can change between planning and execution.
Return errors that are useful without exposing secrets or internal infrastructure. Distinguish invalid input, not authorized, rate limited, unavailable, and policy blocked. Do not let a failed tool call cause the model to silently invent a successful result. Make side effects idempotent. If a request is retried after the network drops, the same action should not send two emails or create two invoices. Attach a stable operation identifier to the action and store the result before allowing a retry to proceed.
For multi-step workflows, checkpoint progress. If an agent has already created a draft and the next step fails, the retry should resume from the known state rather than create a second draft. A useful audit record connects the user request to the actual side effect. Record the request ID, authenticated subject, tenant, tool name, validated argument shape, authorization decision, confirmation state, start and end time, result category, and operation identifier.
Avoid storing secrets and unnecessary personal content. Redact sensitive fields or store references to controlled records instead of copying full prompts into every log.
The audit trail should make these questions answerable:
Without that evidence, a production incident becomes an argument about what the model probably intended.
Add abuse cases to the evaluation set. Ask whether a user can request another tenant’s record by changing an ID. Put an instruction in a retrieved document. Try an unapproved URL. Replay an old confirmation. Send an empty or extremely large recipient list. Retry after a simulated timeout. Remove the user’s role between planning and execution.
Test every tool with a user who can see the interface but should not have the underlying permission. Test a tool with malformed structured output. Test a provider outage and a worker restart.
The strongest result is not “the model followed the prompt.” It is “the application remained safe when the model did not.”
Before shipping an agent tool, confirm that:
This design takes more thought than handing an agent a large toolbox. It also gives builders something far more valuable than a convincing demo: a system that can explain what it is allowed to do, stop when it reaches a boundary, and recover without surprising the user.
If you are building with OTF, use this permission model as part of the production foundation around any AI feature. The model can be replaced. The authorization boundary should remain clear. OWASP Top 10 for Large Language Model Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/