Most developers building LLM applications hit a familiar wall when moving from prototype to production. The app works smoothly during local testing, but in production, responses randomly break expected JSON formats, API calls hang, or retrieval steps yield contextually irrelevant answers.
The default reaction is usually to blame the underlying model, assuming the LLM provider introduced silent regressions. But in most cases, the model isn't the issue.
Prompt engineering only gets an application about 60% to 70% of the way toward reliability. The remaining gap must be filled by standard backend software engineering: strict output validation, explicit timeouts, dynamic retry logic, and context checking.
If your AI agents fail in production, the fix is rarely a rewording of your prompt. It requires building a robust control layer wrapped around the LLM API calls.
LLMs are non-deterministic text generation engines. Relying on prompt instructions alone—like asking the model to "always return valid JSON"—will eventually fail under high traffic or unexpected user input. If your downstream services expect specific keys, missing or renamed fields will break your application pipeline.
Instead of trusting the model's compliance, enforce schema validation at the code level using tools like Pydantic. If validation fails, intercept the error programmatically rather than letting it crash downstream handlers.
import json
from openai import OpenAI
from pydantic import BaseModel, ValidationError
client = OpenAI()
class AgentAction(BaseModel):
thought_process: str
action_name: str
payload: dict
def get_parsed_action(user_input: str) -> AgentAction:
response = client.chat.completions.create(
model="gpt-4o-mini",
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": "Extract the required action. Return JSON matching: thought_process, action_name, payload."},
{"role": "user", "content": user_input}
]
)
raw_content = response.choices[0].message.content
try:
parsed_json = json.loads(raw_content)
return AgentAction(**parsed_json)
except (json.JSONDecodeError, ValidationError) as err:
raise ValueError(f"LLM produced malformed payload: {err}")
Provider APIs experience network degradation, rate limits, and transient server errors. A basic wrapper around an LLM client without timeout bounds or retry mechanisms leads to stalled backend workers or unhandled exceptions exposed to users.
Wrap production calls with exponential backoff and explicit execution timeouts. If the primary model stalls, degrade gracefully to a faster fallback model.
import httpx
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=10),
reraise=True
)
def execute_llm_call(messages: list) -> str:
try:
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
timeout=5.0
)
return response.choices[0].message.content
except (httpx.TimeoutException, Exception) as exc:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
timeout=3.0
)
return response.choices[0].message.content
In Retrieval-Augmented Generation (RAG) systems, developers often attempt to fix inaccurate answers by tweaking prompt phrasing. However, if your vector database returns irrelevant, stale, or poorly chunked context, no system prompt will prevent hallucinations.
Before passing retrieved context to the generation model:
Prompts should not exist as hardcoded, unversioned strings in endpoint definitions. They are system components that evolve over time.
LLM reliability is an infrastructure challenge, not a phrasing exercise. When you wrap your prompts with strong validation, error handling, and retrieval checks, fragile AI scripts become reliable production systems.
Tanmay builds AI agents and automation for businesses and is open to freelance projects. Connect with Tanmay on LinkedIn to discuss building resilient AI systems.