From Vibe Coding to Production Agents: Designing MCP Toolchains That Don’t Hallucinate, Leak, or Break A developer published a design guide for production-grade Model Context Protocol (MCP) toolchains that insert deterministic infrastructure — a policy engine, tool gateway, and sandboxed worker — between an agent's intent and real API calls, treating the model as an untrusted planner rather than a trusted executor. The guide separates agent failures into hallucination, leakage, and breakage classes, each mapped to controls such as closed schemas, scoped credentials, and idempotency, and argues that tool descriptions should be versioned and treated as semi-trusted configuration within the security boundary. Originally published on tamiz.pro https://tamiz.pro/insights/production-mcp-toolchains-agent-security . Vibe coding gets you a working prototype. Production agents require something else entirely: a toolchain where every call is schema-validated, every permission is explicit, every output is verifiable, and every failure is contained. The gap between those two states is not a matter of better prompts. It is an engineering problem involving protocol boundaries, trust zones, execution semantics, and observability. This guide designs a production-grade Model Context Protocol MCP toolchain for agents that operate on real systems. The central principle is simple: treat the model as an untrusted planner, not a trusted executor. A prototype agent typically wires the model directly to tools: php User - LLM - Tool Executor - Real API That topology works until the model invents an argument, passes an unexpected identifier, retries a non-idempotent operation, or retrieves a record containing sensitive information. The model is probabilistic, and its context contains untrusted data. Neither fact changes because the demo succeeded. A production topology inserts deterministic infrastructure between intent and effect: php User - Agent Runtime - Policy Engine - Tool Gateway - Sandboxed Worker - Real API | | | | | | | +-- egress allowlist | | +-- schema validation + redaction | +-- authorization + risk classification +-- trace ID + budget + audit events The model proposes an action. The toolchain decides whether that action is permissible, translates it into a valid operation, executes it with constrained credentials, and records enough evidence to reconstruct what happened. This architecture addresses three distinct failure classes that are often conflated: | Failure class | Example | Primary control | |---|---|---| | Hallucination | The model invents a customer ID or unsupported argument | Closed schemas, enumeration, existence checks | | Leakage | A tool returns secrets or cross-tenant data | Scoped credentials, output filters, taint tracking | | Breakage | A malformed or repeated call corrupts state | Idempotency, transactions, sandboxing, budgets | The important distinction is that “the model was careful” is not a control. Controls belong in code paths the model cannot bypass. MCP separates the model-facing surface from the implementation of capabilities. A server exposes tools, resources, and prompts over a transport such as stdio or HTTP. The client decides which server to connect to and which operations to expose to the model. That separation is useful, but it is not a security boundary by itself. Once a tool is exposed, the model can attempt any operation that tool permits. If a tool accepts arbitrary SQL, the model effectively has arbitrary SQL access. If a tool returns full account records, the model can request data outside the task’s scope. Treat each component according to its trust level: User input: Untrusted Retrieved documents: Untrusted Tool descriptions: Semi-trusted configuration Model output: Untrusted intent Tool implementation: Trusted code, constrained inputs External service data: Untrusted “Semi-trusted configuration” deserves emphasis. Tool descriptions often live in configuration files, repositories, or registries. A compromised description can instruct the model to ignore prior instructions, exfiltrate data through a legitimate parameter, or select a dangerous tool. Descriptions should be reviewed, versioned, and included in the security boundary. The most effective design pattern is to expose narrow, intent-shaped tools rather than generic primitives. Compare: Bad: query database sql: string Good: find invoices customer id: string, status: enum The first tool transfers the entire database query language into the model’s responsibility. The second encodes business constraints in a contract the model cannot alter. A useful heuristic: if a tool requires the model to understand a complex grammar to use it correctly, the tool is too low-level for reliable agentic execution. Production agents frequently need to access multiple systems with different sensitivity levels. Organize servers into trust zones and permit transitions explicitly. Zone A: Read-only knowledge - documentation search - public product catalog Zone B: Internal business data - customer records - support tickets Zone C: State-changing operations - refunds - ticket closure - deployments Moving from Zone A to Zone B may require additional authorization. Moving into Zone C may require approval, a dry run, or a policy decision based on the operation’s risk. A common mistake is exposing all zones to the same model context. The model then needs no special permission to use the most sensitive tool, and any prompt injection can attempt it. Keeping zones separate reduces blast radius and lets the runtime apply different policies. Hallucination in agentic systems is not limited to invented prose. It includes invented identifiers, incorrect enum values, unsupported operations, and assumptions about resource state. Every tool should define a strict input schema with required fields, additional properties rejected, and enums where the domain is finite. type FindInvoiceInput = { customer id: string; status: "draft" | "open" | "paid" | "void"; issued after?: string; }; The runtime should validate against the schema before the implementation runs. Reject unknown properties instead of silently ignoring them. Silently dropping an unexpected field is dangerous because the model believes it specified a constraint that the executor discarded. For example, using Zod: js import { z } from "zod"; export const FindInvoiceInput = z.object { customer id: z.string .regex /^cus a-zA-Z0-9 {12}$/ , status: z.enum "draft", "open", "paid", "void" , issued after: z.string .datetime .optional , } .strict ; Schema validation catches malformed intent. It does not catch plausible but nonexistent identifiers. Do not ask the model to synthesize opaque identifiers. Provide discovery and resolution tools so the agent can obtain valid values through a controlled lookup. js const FindCustomer = z.object { query: z.string .min 2 .max 120 , } .strict ; async function findCustomer input: unknown, ctx: ToolContext { const { query } = FindCustomer.parse input ; const matches = await ctx.customerDirectory.search query, { limit: 5, tenantId: ctx.principal.tenantId, } ; return { matches: matches.map customer = { customer id: customer.id, display name: customer.displayName, } , }; } The response should not expose the full directory. Returning a bounded list of scoped identifiers gives the model what it needs without creating a general-purpose data extraction channel. State-changing tools should communicate what will happen, what state is required, and what the result means. Include a preflight operation for consequential actions: php prepare refund payment id, amount, reason - RefundPlan commit refund plan id, idempotency key - RefundResult The prepare step can verify the payment exists, the amount is within policy, and the reason is allowed. It returns a short-lived, signed plan identifier. The commit step executes the exact operation that was validated. This two-phase pattern prevents a model from improvising the business logic of a destructive operation. It also enables human approval between preparation and execution. Free-form text encourages the model to infer facts that the tool did not provide. Prefer structured responses with explicit nullability and bounded cardinality. type RefundResult = { status: "succeeded" | "rejected" | "pending"; refund id?: string; rejection code?: "amount exceeds limit" | "already refunded" | "payment closed"; }; Use discriminated unions rather than a generic error string. If the tool returns "Payment is not refundable" , the model might reinterpret it. If it returns status: "rejected" and rejection code: "payment closed" , the runtime can enforce deterministic behavior. Limit result sizes at the tool boundary. A tool that can return 100,000 records is both a context-management problem and a leakage vector. The tool gateway is the point where model intent becomes an operation. It should perform the same work for every request: authenticate, authorize, validate, normalize, classify, execute, filter, and audit. Model request | v Authentication Who is asking? | v Authorization May this principal perform this operation? | v Validation Is the payload structurally valid? | v Normalization Convert formats, enforce canonical values | v Risk policy Approval? Dry run? Allowed target? | v Execution Least-privilege worker, bounded resources | v Output filter Redact, truncate, remove disallowed fields | v Audit event Record decision and outcome Models produce inconsistent representations of dates, amounts, and identifiers. Normalize these values before they reach business logic. function normalizeToolInput input: unknown, schema: ZodSchema : NormalizedInput { const parsed = schema.parse input ; return { ...parsed, issued after: parsed.issued after ? new Date parsed.issued after .toISOString : undefined, }; } Do not rely on the downstream service to interpret loose formats. If a downstream API accepts both milliseconds and ISO timestamps, the model must not choose between them. Risk classification should be a property of the tool and operation, not a decision made by the model. js type RiskClass = "read" | "write" | "sensitive read" | "destructive"; const riskByTool: Record