Why Your LLM App Will Fail at 3AM (And How to Build One That Won’t)

LLM applications in production often fail silently by returning confident but hallucinated answers without triggering errors or alerts, a failure mode traditional monitoring tools cannot detect. To build reliable systems, teams must choose models based on workload-specific trade-offs in cost and capability, implement observability that measures response quality rather than just system health, and use orchestration frameworks like LangChain or LlamaIndex with proper retrieval infrastructure such as pgvector.

We shipped our LLM feature on a Tuesday. By Thursday 3AM, it was returning hallucinated answers with 100% confidence. No errors in the logs. No alerts fired. Latency was perfect. The model was working exactly as designed and completely wrong. That’s the failure mode nobody warns you about before you put an LLM in production. Traditional APM tools Datadog, New Relic, Prometheus tell you whether your system worked . LLM observability tells you whether it worked well . Those are not the same thing. A language model can serve a response in 200ms with a 200 status code and still answer the wrong question, cite a deleted document, or quietly ignore your system prompt after a code change touched it. If you’ve already built a RAG pipeline and want the deep-dive on retrieval failures specifically, I covered the 5 ways RAG pipelines break in production here https://medium.com/towards-artificial-intelligence/building-a-rag-pipeline-that-doesnt-fall-apart-1f7dfbb8e1fc . This article goes wider from model selection through to the 3AM runbook. Before any code, the most consequential decision you’ll make is which model to call. In 2026 the choice has stopped being “which is best” and started being “which fits this workload.” Here’s where the frontier models actually specialise: Gemini 2.5 Pro input jumps to ~$2.50/1M once your prompt crosses 200k tokens relevant if you’re stuffing large retrieved context. Every price here is input-only, per 1M tokens; output runs 4–6x higher. Prices move fast check each provider’s pricing page before you commit a budget to a spreadsheet. For most RAG applications: Gemini 2.5 Pro or GPT-5.5. The 1M context window on Gemini means you can stuff significantly more retrieved chunks before hitting limits. For coding assistants or agentic systems: Claude Opus 4.8. It’s the current front-runner on SWE-bench Verified for code reasoning check the live leaderboard before you quote a number, because the top spot changes every few weeks. For high-volume pipelines where cost matters: Gemini Flash 2.5 at $0.30/1M tokens. Two teams building similar applications can end up with 10x different AI costs based purely on which model tier they chose Opus at $5/1M vs Flash at $0.30/1M is nearly a 17x spread on input. At 100M tokens/day, that’s the difference between roughly $15,000/month and $900/month. Run your own numbers before you commit ; the formula below does it in three lines. The budget decision formula: python def monthly ai cost daily requests: int, avg tokens per request: int, price per million: float - float: """Estimate monthly LLM API cost.""" monthly tokens = daily requests avg tokens per request 30 return monthly tokens / 1 000 000 price per million Example: 50,000 requests/day, 2,000 tokens eachprint monthly ai cost 50 000, 2 000, 5.00 Claude Opus: $15,000/moprint monthly ai cost 50 000, 2 000, 1.25 Gemini 2.5 Pro: $3,750/moprint monthly ai cost 50 000, 2 000, 0.30 Gemini Flash: $900/mo Run this before you pick a model. The cost difference is real. In 2026, most production LLM teams use LangChain for orchestration and LlamaIndex for the retrieval layer underneath. Pure RAG over documents → LlamaIndex. RAG + agents + tools → LangChain on top. Here’s a full working application stack. Not a toy snippet — this is close to what we run. python from llama index.core import VectorStoreIndex, SimpleDirectoryReaderfrom llama index.core.node parser import SentenceSplitterfrom llama index.vector stores.postgres import PGVectorStorefrom llama index.embeddings.openai import OpenAIEmbeddingimport psycopg2 Connect to pgvector Postgres with vector extension pgvector is the right call for most teams - no separate infra, ACID guarantees, and it handles 50+ concurrent requests on a single nodeconnection = psycopg2.connect host="localhost", database="llm app", user="postgres", password="your password" vector store = PGVectorStore.from params host="localhost", port=5432, database="llm app", user="postgres", password="your password", table name="document embeddings", embed dim=1536, text-embedding-3-small dimensions Load and chunk documentsdocuments = SimpleDirectoryReader "./docs" .load data Chunk size matters. 512 tokens is the sweet spot for most RAG workloads: too small = retrieval misses context, too large = noise drowns signalnode parser = SentenceSplitter chunk size=512, chunk overlap=50, overlap prevents cutting sentences at boundaries Build the indexembed model = OpenAIEmbedding model="text-embedding-3-small" index = VectorStoreIndex.from documents documents, vector store=vector store, embed model=embed model, transformations= node parser , show progress=True, print f"Indexed {len documents } documents into pgvector" python from langchain openai import ChatOpenAIfrom langchain core.prompts import ChatPromptTemplatefrom langchain core.output parsers import StrOutputParserfrom langchain core.runnables import RunnablePassthroughfrom llama index.core import QueryBundle The system prompt is production code, not a demo prompt. Version it. Track changes. Treat it like a config file.SYSTEM PROMPT = """You are a technical support assistant for our platform.Answer questions using ONLY the context provided below.If the context does not contain the answer, say "I don't have information on that."Do not speculate. Do not use knowledge outside the provided context.Context: {context}"""def get retriever index, similarity top k: int = 4 : """Return a retriever that fetches top-k similar chunks.""" return index.as retriever similarity top k=similarity top k def format docs nodes - str: """Format retrieved nodes into a single context string.""" return "\n\n---\n\n".join node.text for node in nodes Build the chainllm = ChatOpenAI model="gpt-5", or "gemini-2.5-pro" via langchain google genai temperature=0, 0 = deterministic, critical for factual Q&A max tokens=1024, request timeout=30, explicit timeout - never trust the default prompt = ChatPromptTemplate.from messages "system", SYSTEM PROMPT , "human", "{question}" , retriever = get retriever index LCEL chain: question → retrieve → format → prompt → LLM → parsechain = { "context": lambda x: format docs retriever.retrieve x "question" , "question": RunnablePassthrough , } | prompt | llm | StrOutputParser Usageresponse = chain.invoke {"question": "How do I reset my API key?"} print response The code above works in a demo. Here’s what breaks it in production. The model returns a confident, well-formatted answer sourced from a document you deleted 3 days ago. The embedding is still in your vector store. Your deletion pipeline didn’t clean it up. No error. No alert. Latency: 180ms. Status: 200. The fix: hard delete from your vector store on document deletion. python from llama index.vector stores.postgres import PGVectorStoredef delete document embeddings doc id: str, vector store: PGVectorStore : """ Remove all embeddings for a document when it's deleted or updated. Call this BEFORE re-ingesting updated documents. Without this, you get stale embeddings serving outdated answers. """ vector store.delete ref doc id=doc id print f"Deleted embeddings for document: {doc id}" Hook this into your document management system Webhook on document delete → delete document embeddings Your code was refactored. Someone moved the system prompt definition. The new version no longer includes the “only answer from context” instruction. The model starts answering from its training data. Nobody notices for 4 days because the answers sound correct. This happened to us. The answers were plausible but wrong the model was substituting its general knowledge for outdated product-specific context. The fix: version your prompts and test them in CI. prompts.py — treat this like application config, not inline stringsfrom dataclasses import dataclassfrom datetime import datetime@dataclassclass PromptVersion: version: str created at: str template: str notes: strPROMPTS = { "v1.0": PromptVersion version="v1.0", created at="2026-06-01", template="You are a helpful assistant. Context: {context}", notes="Initial version - too permissive, allows hallucination" , "v1.1": PromptVersion version="v1.1", created at="2026-07-01", template="""You are a technical support assistant.Answer ONLY from the provided context.If unsure, say 'I don't have information on that.'Context: {context}""", notes="Added hard constraint - significantly reduced hallucination rate" ,}ACTIVE PROMPT = PROMPTS "v1.1" Your retriever fetches 10 chunks, each 512 tokens. Plus system prompt 300 tokens , user question 50 tokens , conversation history 800 tokens . Total: ~6,250 tokens. But you capped max tokens at 1,024 for the response and you're on a model tier with an 8k context window and nobody did the arithmetic on input + output together. Best case, the API rejects the request with a context-length error and your endpoint 500s. Worse case an older self-hosted model or a middleware layer that truncates for you the last chunks silently fall off the end. Either way the most relevant retrieved context never reaches the model, and the failure shows up as “answers got worse,” not as an error in your logs. python import tiktokendef check context window system prompt: str, context: str, question: str, model: str = "gpt-5", max tokens: int = 8192, - dict: """ Validate total token count before sending to the model. Raise early instead of silently truncating. """ enc = tiktoken.encoding for model "gpt-4" approximation for GPT-5 total = len enc.encode system prompt + len enc.encode context + len enc.encode question buffer = 1024 reserve tokens for the response return { "total tokens": total, "max allowed": max tokens - buffer, "within limit": total <= max tokens - buffer , "overflow by": max 0, total - max tokens - buffer , } Usage - call this before every LLM invocationresult = check context window system prompt=ACTIVE PROMPT.template, context=format docs retrieved nodes , question=user question, if not result "within limit" : Reduce top k or truncate context, don't silently overflow raise ValueError f"Context window overflow: {result 'overflow by' } tokens over limit. " f"Reduce similarity top k or truncate documents." Your service gets a traffic spike. 500 concurrent requests hit your LLM API simultaneously. OpenAI returns 429 rate limit exceeded . Your retry logic has no jitter all 500 requests retry at exactly the same interval. You hit the rate limit again. Repeat until the queue is exhausted or you’ve burned your error budget. python import asyncioimport randomfrom openai import AsyncOpenAI, RateLimitErrorclient = AsyncOpenAI async def call llm with backoff messages: list, model: str = "gpt-5", max retries: int = 4, - str: """ LLM call with exponential backoff + full jitter. Prevents synchronized retry storms under rate limiting. """ for attempt in range max retries : try: response = await client.chat.completions.create model=model, messages=messages, temperature=0, max tokens=1024, return response.choices 0 .message.content except RateLimitError: if attempt == max retries - 1: raise Full jitter: random sleep up to cap Prevents all retries from firing at the same time cap = 60 max 60 second wait sleep time = random.uniform 0, min cap, 1 2 attempt print f"Rate limited. Attempt {attempt + 1}/{max retries}. " f"Waiting {sleep time:.1f}s before retry." await asyncio.sleep sleep time raise RuntimeError "Max retries exceeded on LLM API call" This is where most teams get it completely wrong. They track HTTP status codes and latency, declare it “monitored,” and go home. That’s not observability. That’s infrastructure monitoring. LLM observability requires an entirely different layer. The failure modes above hallucination, prompt drift, context truncation, degraded retrieval quality all look fine on your Datadog latency dashboard. python from prometheus client import Counter, Histogram, Gaugeimport time 1. Latency - time-to-first-token and total generation timeLLM LATENCY = Histogram "llm request duration seconds", "LLM API call duration", "model", "endpoint" , buckets= 0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0 , 2. Token consumption - drives cost and context window awarenessTOKEN USAGE = Counter "llm tokens total", "Total tokens consumed", "model", "token type" , token type: input / output 3. Retrieval quality - are we fetching relevant context?RETRIEVAL SCORE = Histogram "retrieval similarity score", "Cosine similarity of retrieved chunks to query", "index name" , buckets= 0.5, 0.6, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0 , 4. Answer quality - the metric most teams skipANSWER CONFIDENCE = Gauge "llm answer confidence", "LLM judge score for answer quality 0-1 ", "model", "query type" , class InstrumentedLLMClient: """Wraps LLM calls with full observability instrumentation.""" def init self, model: str : self.model = model self.client = AsyncOpenAI async def chat self, messages: list, query type: str = "general" - dict: start = time.time response = await call llm with backoff messages, model=self.model duration = time.time - start Record every call LLM LATENCY.labels model=self.model, endpoint="chat" .observe duration return {"response": response, "latency": duration}The LLM judge: catching hallucinations automatically Standard monitoring tells you nothing about answer quality. The only way to catch hallucinations at scale is to use a second LLM call as a judge, a model that evaluates whether the answer is actually grounded in the retrieved context. python async def evaluate answer grounding question: str, context: str, answer: str, judge model: str = "gpt-5", use a capable model for judging - dict: """ LLM-as-judge: evaluates whether the answer is grounded in the context. Returns a score 0-1 and a reason. Score < 0.7 = likely hallucination. Alert on this. """ judge prompt = f"""You are evaluating whether an AI answer is properly grounded in the provided context.Question: {question}Context provided to the AI:{context}AI Answer:{answer}Evaluate strictly:1. Does the answer use ONLY information from the context? 0.0 = uses outside knowledge, 1.0 = fully grounded 2. Is the answer factually consistent with the context? 0.0 = contradicts context, 1.0 = fully consistent Respond with JSON only:{{"grounding score": 0.0-1.0, "consistency score": 0.0-1.0, "reason": "brief explanation"}}""" response = await client.chat.completions.create model=judge model, messages= {"role": "user", "content": judge prompt} , temperature=0, response format={"type": "json object"}, import json result = json.loads response.choices 0 .message.content Record to Prometheus overall score = result "grounding score" + result "consistency score" / 2 ANSWER CONFIDENCE.labels model=judge model, query type="rag grounding" .set overall score Alert threshold: below 0.7 = probable hallucination if overall score < 0.7: print f"⚠️ Low grounding score: {overall score:.2f}. " f"Reason: {result 'reason' }" return result The cost of LLM judges: Yes, you’re calling the LLM twice per request. At Gemini Flash prices $0.30/1M , judging 10,000 answers/day with a 500-token judge prompt costs about $1.50/day. That’s cheaper than one customer support ticket for a hallucinated answer. Don’t judge every request in production. Sample 5–10%. That’s enough to catch systematic failures before they become incidents. Here’s what the full system looks like: User Request │ ▼┌─────────────────┐│ FastAPI App │ ← Rate limiting, auth, request validation└────────┬────────┘ │ ▼┌─────────────────┐│ Token Budget │ ← Check context window before calling LLM│ Check │└────────┬────────┘ │ ▼┌─────────────────┐ ┌──────────────────┐│ LlamaIndex │────▶│ pgvector ││ Retriever │ │ embeddings │└────────┬────────┘ └──────────────────┘ │ ▼┌─────────────────┐│ LangChain │ ← Versioned prompt + model call with backoff│ Chain LLM │└────────┬────────┘ │ ▼┌─────────────────┐│ LLM Judge │ ← Sample 5-10% of responses for grounding check│ async │└────────┬────────┘ │ ▼┌─────────────────┐│ Prometheus │ ← Latency, tokens, retrieval score, answer quality│ + Grafana │└─────────────────┘ When your LLM application pages you at 3AM, your first 10 minutes look different from any other service. Standard service failure: check error rates → check recent deploys → rollback. LLM application failure: the alert might not be an error rate. It’s a quality degradation that your users noticed before your monitoring did. Step 1: Is this a model provider outage? Check status pages before doing anything else https://status.openai.com https://status.anthropic.com https://status.cloud.google.com Step 2: Check your LLM-specific metrics If you wired up the Prometheus metrics above, you can query: - llm request duration seconds P99 is latency spiking? - llm tokens total rate are requests going through? - llm answer confidence has quality dropped suddenly? - retrieval similarity score is retrieval degraded? Step 3: Sample recent prompt/response pairs Every LLM call should be logged with: prompt, context, response, latency, tokens Pull the last 20 from the past 30 minutes and read them manually This takes 5 minutes and catches what no metric can Step 4: Check for prompt driftgit log --since="48 hours ago" -- prompts.py did the prompt change?git diff HEAD~1 prompts.py what changed? Step 5: Check vector store for stale embeddings If a major document update shipped recently, stale embeddings may be serving bad contextSELECT COUNT , MIN created at , MAX created at FROM document embeddingsWHERE created at < NOW - INTERVAL '7 days'; Everything above works with OpenAI, Anthropic, or Google APIs. If you’re self-hosting Llama 3.3, Mistral Large 2, or DeepSeek V4, the architecture is the same the operational surface expands significantly. Self-hosting only makes financial sense above roughly $50,000/month in API costs. Below that, the GPU infrastructure, the model serving maintenance vLLM or TGI , the reliability engineering, and the upgrade cycle cost more than the API savings. If you do go self-hosted: python Replace the OpenAI client with an Ollama or vLLM endpointfrom openai import AsyncOpenAI vLLM serves an OpenAI-compatible API Just change the base url - everything else stays the sameclient = AsyncOpenAI base url="http://your-vllm-server:8000/v1", api key="not-required-for-local", Same call, different endpointresponse = await client.chat.completions.create model="meta-llama/Llama-3.3-70B-Instruct", your deployed model messages= {"role": "user", "content": "Hello"} , The observability layer stays identical. The failure modes change slightly — you now own model loading failures, GPU OOM errors, and model version management on top of everything above. If you found this article helpful, a clap would help Towards AI https://pub.towardsai.net/?source=your stories outbox---writer outbox drafts----------------------------------------- share it with more engineers building in this space. The next article covers LLM observability in depth : exactly what to put in Grafana dashboards so you can tell the difference between a model provider hiccup and actual quality degradation. If you’re earlier in the journey and building your first RAG pipeline, start with Building a RAG Pipeline That Doesn’t Fall Apart first. Why Your LLM App Will Fail at 3AM And How to Build One That Won’t https://pub.towardsai.net/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont-254d3940a968 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.