Prompt Engineering Isn't Enough: Building Reliable LLM Applications A developer argues that prompt engineering alone only gets LLM applications 60% to 70% of the way to reliability, and that the remaining gap requires standard backend engineering. The writeup recommends enforcing Pydantic schema validation on model output, wrapping API calls with explicit timeouts and exponential backoff via tenacity, falling back to a smaller model when the primary stalls, and checking retrieval context before generation in RAG systems. Most developers building LLM applications hit a familiar wall when moving from prototype to production. The app works smoothly during local testing, but in production, responses randomly break expected JSON formats, API calls hang, or retrieval steps yield contextually irrelevant answers. The default reaction is usually to blame the underlying model, assuming the LLM provider introduced silent regressions. But in most cases, the model isn't the issue. Prompt engineering only gets an application about 60% to 70% of the way toward reliability. The remaining gap must be filled by standard backend software engineering: strict output validation, explicit timeouts, dynamic retry logic, and context checking. If your AI agents fail in production, the fix is rarely a rewording of your prompt. It requires building a robust control layer wrapped around the LLM API calls. LLMs are non-deterministic text generation engines. Relying on prompt instructions alone—like asking the model to "always return valid JSON"—will eventually fail under high traffic or unexpected user input. If your downstream services expect specific keys, missing or renamed fields will break your application pipeline. Instead of trusting the model's compliance, enforce schema validation at the code level using tools like Pydantic. If validation fails, intercept the error programmatically rather than letting it crash downstream handlers. python import json from openai import OpenAI from pydantic import BaseModel, ValidationError client = OpenAI class AgentAction BaseModel : thought process: str action name: str payload: dict def get parsed action user input: str - AgentAction: response = client.chat.completions.create model="gpt-4o-mini", response format={"type": "json object"}, messages= {"role": "system", "content": "Extract the required action. Return JSON matching: thought process, action name, payload."}, {"role": "user", "content": user input} raw content = response.choices 0 .message.content try: parsed json = json.loads raw content return AgentAction parsed json except json.JSONDecodeError, ValidationError as err: Catch schema violations immediately before passed to downstream services raise ValueError f"LLM produced malformed payload: {err}" Provider APIs experience network degradation, rate limits, and transient server errors. A basic wrapper around an LLM client without timeout bounds or retry mechanisms leads to stalled backend workers or unhandled exceptions exposed to users. Wrap production calls with exponential backoff and explicit execution timeouts. If the primary model stalls, degrade gracefully to a faster fallback model. python import httpx from tenacity import retry, stop after attempt, wait exponential @retry stop=stop after attempt 3 , wait=wait exponential multiplier=1, min=2, max=10 , reraise=True def execute llm call messages: list - str: try: Enforce an explicit HTTP timeout response = client.chat.completions.create model="gpt-4o", messages=messages, timeout=5.0 return response.choices 0 .message.content except httpx.TimeoutException, Exception as exc: Catch network hangs and fall back to a smaller payload/model if necessary response = client.chat.completions.create model="gpt-4o-mini", messages=messages, timeout=3.0 return response.choices 0 .message.content In Retrieval-Augmented Generation RAG systems, developers often attempt to fix inaccurate answers by tweaking prompt phrasing. However, if your vector database returns irrelevant, stale, or poorly chunked context, no system prompt will prevent hallucinations. Before passing retrieved context to the generation model: Prompts should not exist as hardcoded, unversioned strings in endpoint definitions. They are system components that evolve over time. LLM reliability is an infrastructure challenge, not a phrasing exercise. When you wrap your prompts with strong validation, error handling, and retrieval checks, fragile AI scripts become reliable production systems. Tanmay builds AI agents and automation for businesses and is open to freelance projects. Connect with Tanmay on LinkedIn to discuss building resilient AI systems.