An AI agent does not need to be malicious to damage production. It only needs the wrong tool, the wrong database, the wrong customer ID, or one confident step that nobody checked.
That is the uncomfortable part of building agentic features: prompts can suggest safe behavior, but they do not enforce it. If your agent can call tools, write records, send emails, run SQL, trigger workflows, or spend money, you need a deterministic layer between the model and the action.
That layer is an AI agent runtime policy system.
Think of it as a security checkpoint for every tool call. The model can propose an action. The policy layer decides whether that action is allowed, denied, modified, delayed for approval, or logged for review.
This guide is for builders shipping AI features with real customer impact. No vendor pitch. Just architecture, checks, schemas, and mistakes to avoid.
AI products are moving from chat boxes to agents that act.
Recent developer signals point in the same direction: agent-first frameworks, AI gateways with spend caps, MCP-style tool registries, human-in-the-loop workflows, and tool authorization experiments. The industry is making it easier to give agents more tools.
That creates a new risk.
Most apps already check what a human user can do. But agent execution is a chain:
user intent -> prompt -> model reasoning -> tool selection -> arguments -> execution -> side effect
A normal permission check near the API endpoint is still required, but it does not answer everything:
Runtime policy answers those questions before execution.
A lot of content explains prompt injection, RAG risks, or broad AI governance. Useful, but builders often need a narrower answer:
"My agent is about to call a tool. What should my app check before that call runs?"
The practical gap is agent tool call authorization, runtime AI policy enforcement, agent permissions, spend caps, and approval gates in one implementation pattern.
Prompt guardrails guide the model. Runtime policy enforces what the system allows.
| Layer | What it does | Weakness |
|---|---|---|
| System prompt | Guides behavior | Can be ignored or manipulated |
| Tool descriptions | Explain actions | Often too broad |
| App authorization | Checks final API access | May miss intent, autonomy, and cost |
| Runtime policy | Evaluates proposed actions before execution | Requires explicit rules |
Use prompts to shape behavior. Use runtime policy to enforce boundaries.
Put one gate between the agent and every tool.
Agent orchestrator
-> proposed tool call
-> runtime policy engine
-> allow / deny / hold / modify
-> tool executor
-> app API, database, queue, or external API
The policy engine should return one of four decisions:
allow
: run the tool call.deny
: block it and return a safe explanation.hold
: for human approval.modify
: reduce scope, mask fields, or route to a safer tool.The agent never calls production tools directly. It requests execution through the gate.
Do not pass raw tool calls straight into your executor. Wrap every proposed action in a server-owned object.
type AgentActionEnvelope = {
action_id: string;
tenant_id: string;
user_id: string;
agent_id: string;
session_id: string;
task_id: string;
tool_name: string;
tool_version: string;
operation: "read" | "write" | "send" | "delete" | "execute";
target_resource: string;
arguments: Record<string, unknown>;
user_intent: string;
autonomy_mode: "draft" | "copilot" | "autopilot";
environment: "dev" | "staging" | "production";
estimated_cost_cents?: number;
risk_flags?: string[];
created_at: string;
};
The model should not decide the security context. Your app creates it.
A good envelope gives policy code enough information to answer: who, which tenant, which tool, which operation, which resource, which mode, which environment, and what cost.
Tool access should not be a simple yes/no flag.
| Tier | Examples | Default decision |
|---|---|---|
| 0: Safe read | Search docs, summarize public pages | Allow with logging |
| 1: Tenant read | Read tickets, invoices, internal notes | Allow if scoped |
| 2: Draft write | Draft reply, prepare report | Allow or hold by mode |
| 3: Reversible write | Add comment, update tag | Allow if scoped |
| 4: External side effect | Send email, call webhook, invite user | Hold unless delegated |
| 5: Destructive or privileged | Delete data, change roles, run production SQL | Deny or require strong approval |
A single tool may expose multiple tiers. For example, a CRM tool can read accounts, update fields, merge contacts, and delete records. Treat each operation separately.
You can start with code. You do not need a heavy rules engine on day one.
type PolicyDecision = {
decision: "allow" | "deny" | "hold" | "modify";
reason: string;
required_approval_role?: "owner" | "admin" | "operator";
policy_ids: string[];
};
function evaluatePolicy(action: AgentActionEnvelope): PolicyDecision {
if (action.environment === "production" && action.operation === "delete") {
return {
decision: "deny",
reason: "Agents cannot delete production resources.",
policy_ids: ["prod-delete-deny"]
};
}
if (action.operation === "send" && action.autonomy_mode !== "autopilot") {
return {
decision: "hold",
reason: "External sends require approval.",
required_approval_role: "operator",
policy_ids: ["send-requires-approval"]
};
}
if ((action.estimated_cost_cents ?? 0) > 50) {
return {
decision: "hold",
reason: "Estimated cost exceeds the per-action budget.",
required_approval_role: "admin",
policy_ids: ["cost-threshold-hold"]
};
}
if (action.risk_flags?.includes("contains_secret")) {
return {
decision: "deny",
reason: "Tool arguments contain a detected secret.",
policy_ids: ["secret-egress-deny"]
};
}
return {
decision: "allow",
reason: "Action passed runtime policy checks.",
policy_ids: ["default-allow-scoped"]
};
}
Keep this boring. Policy should be deterministic, testable, and easy to explain during an incident review.
A user may be an admin. That does not mean every agent running under that user should get admin power.
Use a delegation object:
{
"delegation_id": "del_123",
"tenant_id": "tenant_abc",
"user_id": "user_456",
"agent_id": "support_triage_agent",
"allowed_tools": ["ticket.search", "ticket.comment", "kb.search"],
"allowed_operations": ["read", "draft", "comment"],
"denied_operations": ["delete", "send_refund", "change_role"],
"max_cost_cents": 200,
"expires_at": "2026-07-10T10:30:00Z"
}
Enforce narrowing:
This is similar to OAuth scopes, but it must cover actions, resources, cost, and time windows.
Models produce plausible arguments. Plausible is not safe.
Use schemas and business rules before execution.
import { z } from "zod";
const SendEmailArgs = z.object({
to: z.string().email(),
subject: z.string().min(3).max(120),
body: z.string().min(10).max(5000),
customer_id: z.string().startsWith("cus_"),
attachments: z.array(z.string()).max(3).default([])
});
function validateSendEmailArgs(args: unknown) {
return SendEmailArgs.parse(args);
}
Then add server-side checks:
Argument validation catches both model mistakes and prompt-injection fallout.
A common agent failure is action drift.
The user asks: "Summarize this renewal risk."
The agent decides: "Email the customer with a discount offer."
Helpful? Maybe. Allowed? Not unless the user asked for it.
Store a task contract:
{
"task_id": "task_789",
"intent": "summarize_renewal_risk",
"allowed_outcomes": ["summary", "risk_score", "recommended_next_steps"],
"forbidden_outcomes": ["send_email", "change_price", "update_contract"],
"requires_approval_for": ["external_message", "billing_change"]
}
Block actions that exceed the contract. This keeps agents useful without letting them expand their own authority.
Agent cost is not only tokens. It is scraping, enrichment, browser sessions, vector searches, retries, and external APIs.
Track budgets per task, tenant, user, agent type, and integration.
async function checkBudget(action: AgentActionEnvelope) {
const spent = await getTaskSpend(action.task_id);
const limit = await getTaskBudget(action.task_id);
const estimate = action.estimated_cost_cents ?? 0;
if (spent + estimate > limit) {
return { decision: "hold", reason: "Task budget would be exceeded." };
}
return { decision: "allow" };
}
Cost policy prevents an agent from burning money while looking productive.
Bad approval UX says:
"Approve tool call
send_email
with this JSON?"
Better approval UX says:
"Approve sending this renewal-risk summary to Priya at Acme Corp? No pricing changes, no attachments, no contract edits."
Approval screens should include:
Store approvals as first-class records, not as buried chat history.
Log every decision, not only blocked actions.
Include:
Avoid storing raw sensitive payloads forever. Prefer hashes, redacted previews, and short-retention encrypted payloads.
Start small:
These eight rules catch many real failures.
MCP-style registries make it easier to expose many actions. That makes policy more important.
Keep server-owned metadata for every tool:
{
"tool_name": "crm.update_contact",
"risk_tier": 3,
"operations": ["write"],
"requires_tenant_scope": true,
"requires_approval": false,
"allowed_environments": ["staging", "production"],
"max_calls_per_task": 10,
"data_classes": ["customer_profile", "email"]
}
Do not trust model-facing tool descriptions as the source of truth. They are documentation, not policy.
Example test:
test("agent cannot delete production customer", () => {
const decision = evaluatePolicy({
action_id: "act_test",
tenant_id: "t1",
user_id: "u1",
agent_id: "support_agent",
session_id: "s1",
task_id: "task1",
tool_name: "customer.delete",
tool_version: "1",
operation: "delete",
target_resource: "customer:cus_123",
arguments: { customer_id: "cus_123" },
user_intent: "clean duplicate records",
autonomy_mode: "autopilot",
environment: "production",
created_at: new Date().toISOString()
});
expect(decision.decision).toBe("deny");
});
Policy tests are cheap insurance.
Customer support agent: read tickets, draft replies, add internal notes. Hold refunds, legal statements, and external sends.
Sales research agent: enrich leads and draft outreach. Cap cost per lead and hold sends unless delegated.
Coding agent: edit files and run tests in a sandbox. Deny production database access, force pushes, secret writes, and risky shell commands.
Finance workflow agent: categorize invoices and draft recommendations. Require strong approval for payments, bank detail changes, and vendor notices.
Before an agent executes a tool call, ask:
If the answer is unclear, hold or deny the action.
AI agent runtime policy is a deterministic control layer that evaluates proposed agent actions before they execute. It checks permissions, scope, arguments, risk, cost, approvals, and audit requirements.
No. Prompt guardrails guide the model. Runtime policy enforces what the system allows. Use both, but rely on runtime policy to block dangerous tool calls.
Yes, if agents can touch customer data, production systems, external messages, billing, or paid APIs. Start with deny rules, approval gates, tenant checks, and cost limits.
Normal authorization checks what a user can do. Runtime policy also checks what the agent was delegated to do, whether the action matches intent, which autonomy mode is active, and whether the cost and risk are acceptable.
No. Safe reads can usually run with logging. External sends, destructive changes, billing updates, permission changes, and production operations need stronger controls.
It cannot remove all prompt-injection risk, but it can stop injected instructions from becoming unsafe actions. Even if the model proposes a bad tool call, the policy layer can deny or hold it.
The next step for AI products is not giving agents every tool and hoping the prompt is good enough.
The next step is controlled execution.
Let the model propose. Let the policy decide. Let the logs prove what happened.