Stop Prompting, Start Schemas: The Real Fix for Unreliable AI Agents

A developer argues that schema-constrained generation, not prompt engineering, is the reliable fix for AI agents that emit malformed output. The writeup explains that prompting is only a soft constraint on a probabilistic model, while masking the vocabulary during decoding so the model cannot emit tokens that violate a JSON Schema, Pydantic model, or grammar guarantees valid output. It recommends validating hard and failing loudly rather than retrying or silently falling back to defaults.

Originally published on tamiz.pro https://tamiz.pro/insights/stop-prompting-start-schemas-ai-agents . Every developer who has shipped an AI agent has hit the same wall. You craft a prompt that looks bulletproof. You instruct the model to "return JSON only." You add "do not wrap in markdown." You even threaten it. And then, in production, the agent returns: Here's your result: { "status": "success", "data": { ... } Trailing backticks. A missing brace. A field that's null when the schema says string . The model "understands" the task, but it can't reliably emit the contract you asked for. The fix isn't a better prompt — it's a schema. This is a deep dive into why schema-constrained generation — not prompt engineering — is the real fix for unreliable AI agents. We'll look at how it works under the hood, why it beats prompt-based parsing, and how to build agents that fail loudly, validate hard, and never silently corrupt data. If you're building production agents with LLMs, this is the mental model you need. Prompting is a soft constraint. It tells the model what you want, but it doesn't give the model a mechanism to guarantee it. The model is a next-token predictor sampling from a probability distribution over the entire vocabulary — and at every step, it can choose a token that breaks your contract. When you say "return JSON," you're asking the model to satisfy a grammar it has only statistically learned. The model has seen thousands of JSON examples in training, so it's likely to produce something JSON-shaped. But "likely" is not "guaranteed." The failure modes are predictable and familiar: max tokens limit mid-object, producing invalid JSON. null for required fields. integer comes back as "42" a string , or a list comes back as a single object. Each of these is a distributional failure. The model's weights encode a tendency, not a rule. And this is the key insight: you cannot prompt your way out of a probability distribution. You can nudge the distribution, but you can't force it to respect a grammar. The second problem is that prompt-based approaches defer all the work to post-hoc parsing . You generate text, then you try to parse it with json.loads or a regex. When parsing fails, you either retry the whole generation expensive, non-deterministic or you fall back to a default silently wrong . Both are bad. Retrying is a band-aid; it doesn't fix the root cause. It just rolls the dice again. The root cause is that you're asking a probabilistic system to produce a deterministic artifact, and you're only enforcing the contract after the fact. Schemas flip this around. Instead of generating freely and hoping, you constrain the generation process itself so that the model physically cannot emit invalid output. That's a fundamentally different architecture. A schema is a formal specification of the output's shape: its fields, types, allowed values, nesting, and constraints. JSON Schema is the canonical example, but the same idea applies to Pydantic models, TypeScript interfaces, Zod schemas, or even a context-free grammar. The fix is to feed that schema into the generation loop , not into the prompt. Instead of letting the model pick any token from the vocabulary at each step, you mask the vocabulary to only the tokens that keep the output valid under the schema. This is called constrained decoding , structured outputs , or grammar-constrained generation . It's the difference between asking a model to "draw a circle" and handing it a compass. With a compass, the circle is guaranteed — the tool enforces the constraint. Here's the conceptual shift: | Approach | Constraint location | Failure mode | Cost of failure | |---|---|---|---| | Prompt-only | Prompt text soft | Malformed output, parsing errors | High — retry, or silent corruption | | Prompt + retry | Prompt + post-hoc parse | Still fails on hard cases | Medium — repeated calls, latency | | Schema-constrained | Generation loop hard | Rare — schema violations mostly impossible | Low — deterministic, one shot | With schema-constrained generation, the contract is enforced during decoding. The model can only choose tokens that advance toward a valid completion. You get what you asked for, every time. To understand why this is so much more reliable, you need to know what's happening at the token level. Let's walk through the mechanics. At every decoding step, the model produces a probability distribution over the vocabulary — a vector of logits, one per token. Normally, you sample from this full distribution. With constrained decoding, you build a mask of allowed tokens and set the logits of disallowed tokens to -infinity or a very large negative number , so they can never be sampled. python python Conceptual — not runnable as-is, but shows the mechanism. import torch logits = model input ids shape: 1, vocab size allowed mask = build allowed token mask state, schema shape: 1, vocab size masked logits = logits.masked fill ~allowed mask, float '-inf' next token = sample masked logits The mask is the entire trick. It's not a heuristic; it's a hard constraint. The model cannot produce a token that would violate the schema, because that token has zero probability. But how do you know which tokens are allowed at each step? You need to track the current state of the partial output and derive the allowed tokens from it. This is where the schema becomes a finite-state machine or a pushdown automaton for nested structures . Imagine the schema: json { "type": "object", "properties": { "name": { "type": "string" }, "age": { "type": "integer", "minimum": 0 } }, "required": "name", "age" } After the model emits { "name": "Alice", , the parser state knows: name is already satisfied. age since required includes it and it's not yet present . "age" or whitespace, or } if The state machine encodes all the grammar rules of JSON Schema: object key ordering or unordered keys , string escapes, number formatting, enum membership, array lengths, minItems / maxItems , additionalProperties , and so on. As the model generates, the state advances, and the allowed token set is recomputed from the new state. For arbitrary JSON Schema, implementations typically compile the schema into a context-free grammar CFG and then use a parser like a GLR or Earley parser to track the valid continuations. Libraries such as Outlines https://dottxt-ai.github.io/outlines/latest/ and Instructor https://python.useinstructor.com/ do exactly this: they convert a Pydantic model or JSON Schema into a grammar, then run constrained decoding against any compatible backend OpenAI, Anthropic, local models via transformers , llama.cpp , vLLM , etc. . python Outlines — schema-constrained generation against a local model. import outlines from pydantic import BaseModel class User BaseModel : name: str age: int model = outlines.models.transformers "meta-llama/Meta-Llama-3-8B-Instruct" generator = outlines.generate.json model, User result = generator "Extract the user from: Alice is 30 years old." print result User name='Alice', age=30 — always valid, no parsing needed The key takeaway: the schema is not documentation. It is the generator. The model never has a chance to emit invalid JSON, because invalid tokens are unreachable. Since late 2024, major providers have shipped this natively. OpenAI's Structured Outputs and Anthropic's tool-use with strict schemas both implement constrained decoding on their side. You pass a JSON Schema, and the API guarantees the response matches it. This is the easiest adoption path for most teams. python python from openai import OpenAI from pydantic import BaseModel client = OpenAI class User BaseModel : name: str age: int response = client.chat.completions.create model="gpt-4o", response format={ "type": "json schema", "json schema": { "name": "user extraction", "strict": True, "schema": User.model json schema } }, messages= {"role": "user", "content": "Alice is 30."} user = User.model validate json response.choices 0 .message.content Notice what changed: there's no json.loads call, no try/except, no regex cleanup, no retry loop. The response is guaranteed to be valid JSON matching the schema, so you can deserialize it directly. If the API can't satisfy the schema, it returns an error — it doesn't silently return garbage. Before you build, you need to pick your schema source of truth. The choice matters because it affects how you define, validate, and version your contracts. | Layer | Pros | Cons | Best for | |---|---|---|---| | JSON Schema raw | Universal, provider-native, language-agnostic | Verbose, no type safety in code | Cross-language contracts, API specs | | Pydantic Python | Type-safe, auto-generates JSON Schema, integrates with FastAPI | Python-only | Python services, FastAPI backends | | TypeScript / Zod | Type-safe, compile-time checks | No direct JSON Schema in all providers | TS frontends, edge functions | | Function calling | Native to providers, supports multi-tool | Less expressive than full JSON Schema | Tool-using agents, ReAct loops | My recommendation for most teams: define your contract once as a Pydantic model or Zod schema , and derive the JSON Schema from it. This gives you type safety in code and a provider-compatible schema for constrained decoding. You never hand-write JSON Schema by hand — that's how drift happens. python python from pydantic import BaseModel, Field from typing import Literal class Order BaseModel : order id: str = Field description="Unique order identifier" status: Literal "pending", "paid", "shipped", "cancelled" total cents: int = Field ge=0, description="Total in cents, non-negative" items: list str = Field min length=1, max length=50 Now Order.model json schema gives you a JSON Schema that both OpenAI and Anthropic can consume directly. And Order.model validate json gives you a validated Python object on the way out. One source of truth, two directions. Let's build a concrete, production-shaped example: an order-processing agent that extracts structured data from a natural-language email and decides whether to approve it. This is the kind of task that breaks naive prompt-based agents constantly. python python from pydantic import BaseModel, Field from typing import Literal, Optional class OrderItem BaseModel : sku: str quantity: int = Field ge=1, le=1000 unit price cents: int = Field ge=0 class Extraction BaseModel : """Extracted order data from a customer email.""" customer email: str order items: list OrderItem = Field min length=1 shipping address: str priority: Literal "standard", "express" = "standard" notes: Optional str = None python python from openai import OpenAI client = OpenAI SYSTEM PROMPT = """ You are an order extraction agent. Extract order information from customer emails. Return ONLY the structured data. Never invent fields that are not in the schema. If a required field is missing from the email, use your best judgment to infer it, but never fabricate a customer email address. """ def extract order email body: str - Extraction: response = client.chat.completions.create model="gpt-4o", response format={ "type": "json schema", "json schema": { "name": "order extraction", "strict": True, "schema": Extraction.model json schema } }, messages= {"role": "system", "content": SYSTEM PROMPT}, {"role": "user", "content": email body} , temperature=0, deterministic, best for extraction No json.loads , no try/except, no regex cleanup. The API guarantees this matches Extraction's schema. return Extraction.model validate json response.choices 0 .message.content Even with structured outputs, you should still validate. Why? Two reasons: 1 it catches semantic issues that JSON Schema can't express, and 2 it protects you if you ever switch to a provider that doesn't support strict schemas. Validation is cheap; silent corruption is expensive. python class ExtractionError Exception : """Raised when an extraction fails semantic validation.""" def init self, message: str, details: dict : super . init message self.details = details def validate semantics extraction: Extraction - None: """Business rules that JSON Schema can't enforce.""" if not extraction.customer email.endswith "@" : raise ExtractionError "Customer email is malformed", { "field": "customer email", "value": extraction.customer email } total = sum i.unit price cents i.quantity for i in extraction.order items if total <= 0: raise ExtractionError "Order total must be positive", { "field": "order items", "value": total } Now the full agent loop. Notice the retry logic is not for parsing failures — those are impossible now. It's for semantic failures and provider errors, which is a much smaller and more predictable surface. python python import logging from tenacity import retry, stop after attempt, wait exponential logger = logging.getLogger name @retry stop=stop after attempt 3 , wait=wait exponential multiplier=1, min=1, max=10 , reraise=True, def safe extract email body: str - Extraction: extraction = extract order email body validate semantics extraction return extraction Usage try: order = safe extract "Hi, please ship 2x SKU-1234 and 1x SKU-5678 to 123 Main St." print f"Processed order for {order.customer email}: {len order.order items } items" except ExtractionError as e: logger.error "Semantic validation failed: %s", e.details Route to human review, don't silently drop raise This is the real fix. The prompt still matters — it guides the model's reasoning — but it no longer bears the burden of enforcing the format . That burden is on the schema, where it belongs. JSON schemas are just the beginning. The same principle extends to tool calling and multi-step agent loops , which is where most production agents actually live. When an agent calls a tool, it's not just returning data — it's selecting an action . That selection is a structured decision, and it should be schema-constrained too. Provider-native tool calling already does this: you declare each tool's parameters as a JSON Schema, and the model must emit a valid call. python tools = { "type": "function", "function": { "name": "get order status", "description": "Get the current status of an order.", "parameters": { "type": "object", "properties": { "order id": {"type": "string", "description": "The order ID"} }, "required": "order id" , "additionalProperties": False } } } With strict: True and additionalProperties: False , the provider guarantees the tool call is well-formed. The agent can't call get order status without an order id , and it can't pass extra parameters. This eliminates an entire class of "tool call hallucination" bugs. A ReAct Reasoning + Acting agent interleaves reasoning steps and tool calls. The classic bug in naive ReAct implementations is that the model emits a malformed action, the parser fails, and the whole loop crashes. Schema enforcement fixes this at the source. python python from pydantic import BaseModel from typing import Union class Think BaseModel : thought: str class Act BaseModel : tool name: str tool args: dict class ReActStep BaseModel : """One step in the ReAct loop. The model must choose exactly one action.""" step: Union Think, Act = Field description="Either a reasoning step or a tool action" done: bool = Field description="True if the task is complete" By making step a discriminated union, you force the model to produce either a thought or an action — not both, not neither, not a malformed hybrid. The loop then becomes deterministic: php python def run agent task: str - str: history = {"role": "user", "content": task} for in range MAX STEPS : response = client.chat.completions.create model="gpt-4o", response format={ "type": "json schema", "json schema": { "name": "react step", "strict": True, "schema": ReActStep.model json schema } }, messages=history, step = ReActStep.model validate json response.choices 0 .message.content if step.done: return final answer step if isinstance step.step, Act : result = execute tool step.step.tool name, step.step.tool args else: result = None reasoning step, no tool call history.append {"role": "assistant", "content": response.choices 0 .message.content} history.append {"role": "user", "content": str result or step.step.thought } raise MaxStepsExceeded Notice there's no json.loads , no if "tool" in text heuristic, no regex extraction. The schema does all the structural work. The model's reasoning is still free — that's what the prompt guides — but the structure is locked. Schemas make agents reliable, but reliability in production requires more than just enabling structured outputs. Here's what I've learned shipping these systems. A schema is an API contract. When you change it, you break every stored response and every downstream consumer. Version it explicitly: python class ExtractionV2 BaseModel : """V2: adds currency field and drops legacy notes field.""" customer email: str order items: list OrderItem shipping address: str currency: Literal "USD", "EUR", "GBP" = "USD" priority: Literal "standard", "express" = "standard" Keep the old version around for replay and migration. Never silently mutate a schema in place. The worst thing an agent can do is return null for a required field and have your code treat it as valid. Always validate on the way out, even with structured outputs. If validation fails, raise an error that gets routed to a human — don't log and continue. python try: extraction = safe extract email except ExtractionError as e: Route to human review queue, not to a silent default review queue.enqueue email, error=e.details raise Extraction is a deterministic task. temperature=0 gives you the most reproducible results. For open-ended reasoning or creative tasks, you might want a bit more variety — but even then, keep the structure constrained and vary the content , not the shape. Track these metrics, and you'll see the difference schemas make: JSON Schema can't express everything. It can't say "this date must be in the future" or "this email must exist in our database" or "these two fields must be consistent with each other." Those are semantic constraints, and they belong in your validation layer Pydantic validators, business rules, etc. . Schema-constrained generation handles syntax ; your code handles semantics . Don't conflate the two. Let's be honest about the limits. Schema-constrained generation is a massive upgrade, but it's not a silver bullet. There are two classes of failure it doesn't fix: Reasoning failures. The model produces perfectly valid JSON, but the content is wrong. It extracts the wrong date, picks the wrong enum value, or infers a fact that isn't in the source. Schema constraints can't fix bad reasoning — they can only make sure the bad reasoning is well-formatted . This is where you need better prompts, better context, retrieval, or a more capable model. Semantic inconsistency. Two fields that are individually valid but jointly impossible e.g., total cents doesn't match the sum of items . No schema can express cross-field invariants, so you need validation logic and, when it fails, a recovery strategy: retry with the validation error fed back into the prompt, or escalate to a human. The schema guarantees the shape . It does not guarantee the truth . Those are two different problems, and you need both. For failure 2, a good pattern is to feed validation errors back into the model as a corrective message: python python def extract with self correction email body: str, max attempts: int = 2 - Extraction: messages = {"role": "user", "content": email body} for attempt in range max attempts : response = client.chat.completions.create model="gpt-4o", response format={"type": "json schema", "json schema": {...}}, messages=messages, try: extraction = Extraction.model validate json response.choices 0 .message.content validate semantics extraction return extraction except ValidationError, ExtractionError as e: if attempt == max attempts - 1: raise messages.append {"role": "assistant", "content": response.choices 0 .message.content} messages.append {"role": "user", "content": f"Your output failed validation: {e}. Please correct it."} This keeps the format guaranteed by the schema, while giving the model a chance to fix its reasoning . It's the best of both worlds. If you're using a provider-native structured outputs API e.g., response format with json schema , malformed JSON should be impossible — the provider enforces it during decoding. If you still see it, you're likely a not passing strict: True , b not setting additionalProperties: False on all objects, or c using a provider that only suggests rather than enforces the schema. Some providers return a schema that's a subset of JSON Schema and silently degrade. Always validate on the way out as a safety net. Function calling is a specific form of structured output where the schema describes a tool's parameters, and the model's output is a tool invocation. Structured outputs is the broader concept: any JSON Schema can constrain the response. Function calling is often the easiest way to get structured outputs on older models, but the dedicated json schema response format is now more general and more reliable. Use Pydantic or Zod in TypeScript as your source of truth and derive JSON Schema from it. Hand-writing JSON Schema is error-prone and leads to drift between your code types and your API contract. With Pydantic, you get type safety, validation, and auto-generated schemas in one place. Only use raw JSON Schema when you need a language-agnostic contract shared across services. If you're building agents that touch production data, make the schema the contract — not the prompt. The prompt guides reasoning; the schema enforces structure. That division of labor is what separates a demo from a system you can actually ship. For more on engineering reliable AI systems, see Tamiz's Insights https://tamiz.pro/insights and the engineering guides at tamiz.pro https://tamiz.pro . Validation is only the first layer of the schema-first approach. The real power emerges when you treat schemas as composable building blocks rather than isolated constraints. Think of it this way: a single schema tells you what a valid response looks like, but a composition of schemas tells you how a system should behave across multiple turns, tools, and failure modes. Let's move from simple validation to a more sophisticated pattern—schema-driven state machines. Consider a customer support agent that needs to handle a multi-step refund process. Instead of writing a prompt that says "handle the refund flow," you define a schema for each state and a transition map between them. python python from pydantic import BaseModel, Field from typing import Literal, Union from enum import Enum class RefundState str, Enum : AWAITING ORDER ID = "awaiting order id" AWAITING REASON = "awaiting reason" AWAITING APPROVAL = "awaiting approval" COMPLETED = "completed" class OrderIDRequest BaseModel : intent: Literal "provide order id" = "provide order id" order id: str = Field ..., pattern=r"^ORD-\d{6}$" class RefundReasonRequest BaseModel : intent: Literal "provide reason" = "provide reason" reason: str = Field ..., min length=10, max length=500 urgency: Literal "low", "medium", "high" = "medium" class ApprovalDecision BaseModel : intent: Literal "approve refund" = "approve refund" approved: bool notes: str | None = None class RefundResponse BaseModel : state: RefundState data: Union OrderIDRequest, RefundReasonRequest, ApprovalDecision Define the transition function def next state current: RefundState, response: RefundResponse - RefundState: if current == RefundState.AWAITING ORDER ID: if isinstance response.data, OrderIDRequest : return RefundState.AWAITING REASON raise ValueError "Expected order ID" if current == RefundState.AWAITING REASON: if isinstance response.data, RefundReasonRequest : return RefundState.AWAITING APPROVAL raise ValueError "Expected refund reason" if current == RefundState.AWAITING APPROVAL: if isinstance response.data, ApprovalDecision : return RefundState.COMPLETED if response.data.approved else RefundState.AWAITING REASON raise ValueError "Expected approval decision" return RefundState.COMPLETED This pattern gives you something that pure prompting can never provide: provable state transitions . You can unit-test every path, enforce invariants at compile time, and guarantee that the agent cannot skip a step or hallucinate a state that doesn't exist. Now that you have schemas, you need an execution layer that actually uses them. The naive approach is to parse the model output and hope it matches. The robust approach is to validate, retry, and fall back through a structured pipeline. Here's a production-ready agent loop that treats schemas as first-class citizens: python python import json from typing import Type, TypeVar, Generic from pydantic import BaseModel, ValidationError T = TypeVar 'T', bound=BaseModel class SchemaAgent Generic T : def init self, model: callable, LLM callable: prompt, schema - str schema: Type T , max retries: int = 3, fallback: T | None = None, : self.model = model self.schema = schema self.max retries = max retries self.fallback = fallback def run self, prompt: str, context: dict | None = None - T: last error = None for attempt in range self.max retries : try: Pass the schema as a structured constraint, not just a prompt hint raw output = self.model prompt=prompt, schema=self.schema.model json schema , context=context or {}, Validate and coerce validated = self.schema.model validate json raw output return validated except ValidationError as e: last error = e Feed the error back into the prompt for self-correction prompt = self. augment prompt prompt, e continue except json.JSONDecodeError as e: last error = e prompt = self. augment prompt prompt, "Response was not valid JSON" continue if self.fallback is not None: return self.fallback raise RuntimeError f"Agent failed after {self.max retries} attempts. Last error: {last error}" def augment prompt self, prompt: str, error: Exception - str: return f""" {prompt} Your previous response was invalid for the following reason: {error} Please provide a corrected response that strictly adheres to the schema. """ Notice what's happening here. The schema isn't just appended to the prompt as text—it's passed as a structured parameter to the model. This forces the model to generate output that can be parsed and validated deterministically. The retry loop includes the validation error in the next prompt, which gives the model a concrete correction target rather than a vague "try again." Real-world agents rarely do one thing. They orchestrate tools, call external APIs, and maintain conversational context. The schema-first approach shines in these scenarios because it gives you a way to define the contract for each step. Here's a pattern for a multi-step agent that handles a complete user request: python python from typing import List, Dict, Any from pydantic import BaseModel class ToolCall BaseModel : tool name: str arguments: Dict str, Any class AgentStep BaseModel : step id: int tool calls: List ToolCall expected output schema: Dict str, Any next step: int | None = None class WorkflowDefinition BaseModel : steps: Dict int, AgentStep entry point: int = 0 class WorkflowExecutor: def init self, workflow: WorkflowDefinition, tools: Dict str, callable : self.workflow = workflow self.tools = tools def execute self, initial input: Dict str, Any - Dict str, Any : current step = self.workflow.entry point context = initial input while current step is not None: step = self.workflow.steps current step Execute each tool call in sequence for tool call in step.tool calls: if tool call.tool name not in self.tools: raise ValueError f"Unknown tool: {tool call.tool name}" result = self.tools tool call.tool name tool call.arguments Validate the result against the expected schema self. validate result result, step.expected output schema Merge result into context context.update result current step = step.next step return context def validate result self, result: Any, schema: Dict str, Any - None: Use a lightweight validator here, or integrate with Pydantic This is where you'd enforce the schema contract pass This pattern is especially valuable when you're building agents that call external APIs. Instead of hoping the LLM correctly formats a function call, you define the exact schema for each tool's input and output. The workflow executor enforces those schemas at runtime, so a malformed tool call fails fast rather than propagating through the system. One of the most underrated benefits of schema-first design is that it gives you a structured way to handle errors. When an agent produces invalid output, you don't just retry—you can analyze why it failed and take corrective action. python python from pydantic import BaseModel, ValidationError from typing import Literal, Union class RecoveryAction BaseModel : action: Literal "retry", "clarify", "escalate" message: str suggested schema: Dict str, Any | None = None class ErrorAnalyzer: def analyze self, error: ValidationError, attempts: int - RecoveryAction: if attempts < 2: return RecoveryAction action="retry", message="Please correct the output to match the schema.", suggested schema=error.schema, if self. is ambiguity error error : return RecoveryAction action="clarify", message="The request is ambiguous. Please provide more specific information.", return RecoveryAction action="escalate", message="Unable to produce valid output. Escalating to human review.", def is ambiguity error self, error: ValidationError - bool: Heuristic: ambiguity errors often involve missing fields or wrong types return any "missing" in str e or "type" in str e .lower for e in error.errors This error analyzer gives you a decision tree for handling failures. Instead of blindly retrying, you can escalate to a human when the model consistently fails, or ask for clarification when the input is ambiguous. This is the kind of robustness that makes agents actually deployable in production. We started this article with a provocative claim: prompting is not the fix for unreliable AI agents. By now, I hope you see why. Prompts are suggestions; schemas are contracts. Suggestions can be ignored; contracts must be honored. The shift to schema-first development changes how you build agents in fundamental ways: The tools I've shown here—Pydantic for validation, state machines for workflow control, and structured error recovery—are not exotic. They're the same patterns you'd use for any production software. The only difference is that now you're applying them to systems that generate code and text dynamically. Stop prompting. Start schemas. Your agents will thank you.