{"slug": "why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont", "title": "Why Your LLM App Will Fail at 3AM (And How to Build One That Won’t)", "summary": "LLM applications in production often fail silently by returning confident but hallucinated answers without triggering errors or alerts, a failure mode traditional monitoring tools cannot detect. To build reliable systems, teams must choose models based on workload-specific trade-offs in cost and capability, implement observability that measures response quality rather than just system health, and use orchestration frameworks like LangChain or LlamaIndex with proper retrieval infrastructure such as pgvector.", "body_md": "We shipped our LLM feature on a Tuesday. By Thursday 3AM, it was returning hallucinated answers with 100% confidence. No errors in the logs. No alerts fired. Latency was perfect. The model was working exactly as designed and completely wrong.\n\nThat’s the failure mode nobody warns you about before you put an LLM in production.\n\nTraditional APM tools Datadog, New Relic, Prometheus tell you whether your system *worked*. LLM observability tells you whether it worked *well*. Those are not the same thing. A language model can serve a response in 200ms with a 200 status code and still answer the wrong question, cite a deleted document, or quietly ignore your system prompt after a code change touched it.\n\nIf you’ve already built a RAG pipeline and want the deep-dive on retrieval failures specifically, I covered the 5 ways [RAG pipelines break in production here](https://medium.com/towards-artificial-intelligence/building-a-rag-pipeline-that-doesnt-fall-apart-1f7dfbb8e1fc). This article goes wider from model selection through to the 3AM runbook.\n\nBefore any code, the most consequential decision you’ll make is which model to call. In 2026 the choice has stopped being “which is best” and started being “which fits this workload.”\n\nHere’s where the frontier models actually specialise:\n\n* Gemini 2.5 Pro input jumps to ~$2.50/1M once your prompt crosses 200k tokens relevant if you’re stuffing large retrieved context. Every price here is input-only, per 1M tokens; output runs 4–6x higher. Prices move fast check each provider’s pricing page before you commit a budget to a spreadsheet.\n\nFor most RAG applications: Gemini 2.5 Pro or GPT-5.5. The 1M context window on Gemini means you can stuff significantly more retrieved chunks before hitting limits.\n\nFor coding assistants or agentic systems: Claude Opus 4.8. It’s the current front-runner on SWE-bench Verified for code reasoning check the live leaderboard before you quote a number, because the top spot changes every few weeks.\n\nFor high-volume pipelines where cost matters: Gemini Flash 2.5 at $0.30/1M tokens. Two teams building similar applications can end up with 10x different AI costs based purely on which model tier they chose Opus at $5/1M vs Flash at $0.30/1M is nearly a 17x spread on input. At 100M tokens/day, that’s the difference between roughly $15,000/month and $900/month. **Run your own numbers before you commit**; the formula below does it in three lines.\n\nThe budget decision formula:\n\n``` python\ndef monthly_ai_cost(daily_requests: int, avg_tokens_per_request: int, price_per_million: float) -> float:    \"\"\"Estimate monthly LLM API cost.\"\"\"    monthly_tokens = daily_requests * avg_tokens_per_request * 30    return (monthly_tokens / 1_000_000) * price_per_million# Example: 50,000 requests/day, 2,000 tokens eachprint(monthly_ai_cost(50_000, 2_000, 5.00))    # Claude Opus:    $15,000/moprint(monthly_ai_cost(50_000, 2_000, 1.25))    # Gemini 2.5 Pro: $3,750/moprint(monthly_ai_cost(50_000, 2_000, 0.30))    # Gemini Flash:   $900/mo\n```\n\nRun this before you pick a model. The **cost** difference is real.\n\nIn 2026, most production LLM teams use **LangChain** for orchestration and **LlamaIndex** for the retrieval layer underneath.\n\n*Pure RAG over documents → LlamaIndex. RAG + agents + tools → LangChain on top.*\n\nHere’s a full working application stack. Not a toy snippet — this is close to what we run.\n\n``` python\nfrom llama_index.core import VectorStoreIndex, SimpleDirectoryReaderfrom llama_index.core.node_parser import SentenceSplitterfrom llama_index.vector_stores.postgres import PGVectorStorefrom llama_index.embeddings.openai import OpenAIEmbeddingimport psycopg2# Connect to pgvector (Postgres with vector extension)# pgvector is the right call for most teams - no separate infra,# ACID guarantees, and it handles 50+ concurrent requests on a single nodeconnection = psycopg2.connect(    host=\"localhost\",    database=\"llm_app\",    user=\"postgres\",    password=\"your_password\")vector_store = PGVectorStore.from_params(    host=\"localhost\",    port=5432,    database=\"llm_app\",    user=\"postgres\",    password=\"your_password\",    table_name=\"document_embeddings\",    embed_dim=1536,  # text-embedding-3-small dimensions)# Load and chunk documentsdocuments = SimpleDirectoryReader(\"./docs\").load_data()# Chunk size matters. 512 tokens is the sweet spot for most RAG workloads:# too small = retrieval misses context, too large = noise drowns signalnode_parser = SentenceSplitter(    chunk_size=512,    chunk_overlap=50,  # overlap prevents cutting sentences at boundaries)# Build the indexembed_model = OpenAIEmbedding(model=\"text-embedding-3-small\")index = VectorStoreIndex.from_documents(    documents,    vector_store=vector_store,    embed_model=embed_model,    transformations=[node_parser],    show_progress=True,)print(f\"Indexed {len(documents)} documents into pgvector\")\npython\nfrom langchain_openai import ChatOpenAIfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_core.output_parsers import StrOutputParserfrom langchain_core.runnables import RunnablePassthroughfrom llama_index.core import QueryBundle# The system prompt is production code, not a demo prompt.# Version it. Track changes. Treat it like a config file.SYSTEM_PROMPT = \"\"\"You are a technical support assistant for our platform.Answer questions using ONLY the context provided below.If the context does not contain the answer, say \"I don't have information on that.\"Do not speculate. Do not use knowledge outside the provided context.Context: {context}\"\"\"def get_retriever(index, similarity_top_k: int = 4):    \"\"\"Return a retriever that fetches top-k similar chunks.\"\"\"    return index.as_retriever(similarity_top_k=similarity_top_k)def format_docs(nodes) -> str:    \"\"\"Format retrieved nodes into a single context string.\"\"\"    return \"\\n\\n---\\n\\n\".join([node.text for node in nodes])# Build the chainllm = ChatOpenAI(    model=\"gpt-5\",          # or \"gemini-2.5-pro\" via langchain_google_genai    temperature=0,          # 0 = deterministic, critical for factual Q&A    max_tokens=1024,    request_timeout=30,     # explicit timeout - never trust the default)prompt = ChatPromptTemplate.from_messages([    (\"system\", SYSTEM_PROMPT),    (\"human\", \"{question}\"),])retriever = get_retriever(index)# LCEL chain: question → retrieve → format → prompt → LLM → parsechain = (    {        \"context\": lambda x: format_docs(retriever.retrieve(x[\"question\"])),        \"question\": RunnablePassthrough(),    }    | prompt    | llm    | StrOutputParser())# Usageresponse = chain.invoke({\"question\": \"How do I reset my API key?\"})print(response)\n```\n\nThe code above works in a demo. Here’s what breaks it in production.\n\nThe model returns a confident, well-formatted answer sourced from a document you deleted 3 days ago. The embedding is still in your vector store. Your deletion pipeline didn’t clean it up.\n\nNo error. No alert. Latency: 180ms. Status: 200.\n\nThe fix: hard delete from your vector store on document deletion.\n\n``` python\nfrom llama_index.vector_stores.postgres import PGVectorStoredef delete_document_embeddings(doc_id: str, vector_store: PGVectorStore):    \"\"\"    Remove all embeddings for a document when it's deleted or updated.    Call this BEFORE re-ingesting updated documents.    Without this, you get stale embeddings serving outdated answers.    \"\"\"    vector_store.delete(ref_doc_id=doc_id)    print(f\"Deleted embeddings for document: {doc_id}\")# Hook this into your document management system# Webhook on document delete → delete_document_embeddings()\n```\n\nYour code was refactored. Someone moved the system prompt definition. The new version no longer includes the “only answer from context” instruction. The model starts answering from its training data. Nobody notices for 4 days because the answers *sound* correct.\n\nThis happened to us. The answers were plausible but wrong the model was substituting its general knowledge for outdated product-specific context.\n\nThe fix: version your prompts and test them in CI.\n\n```\n# prompts.py — treat this like application config, not inline stringsfrom dataclasses import dataclassfrom datetime import datetime@dataclassclass PromptVersion:    version: str    created_at: str    template: str    notes: strPROMPTS = {    \"v1.0\": PromptVersion(        version=\"v1.0\",        created_at=\"2026-06-01\",        template=\"You are a helpful assistant. Context: {context}\",        notes=\"Initial version - too permissive, allows hallucination\"    ),    \"v1.1\": PromptVersion(        version=\"v1.1\",        created_at=\"2026-07-01\",        template=\"\"\"You are a technical support assistant.Answer ONLY from the provided context.If unsure, say 'I don't have information on that.'Context: {context}\"\"\",        notes=\"Added hard constraint - significantly reduced hallucination rate\"    ),}ACTIVE_PROMPT = PROMPTS[\"v1.1\"]\n```\n\nYour retriever fetches 10 chunks, each 512 tokens. Plus system prompt (300 tokens), user question (50 tokens), conversation history (800 tokens).\n\nTotal: ~6,250 tokens. But you capped max_tokens at 1,024 for the response and you're on a model tier with an 8k context window and nobody did the arithmetic on input + output together.\n\nBest case, the API rejects the request with a context-length error and your endpoint 500s. Worse case an older self-hosted model or a middleware layer that truncates for you the last chunks silently fall off the end. Either way the most relevant retrieved context never reaches the model, and the failure shows up as “answers got worse,” not as an error in your logs.\n\n``` python\nimport tiktokendef check_context_window(    system_prompt: str,    context: str,    question: str,    model: str = \"gpt-5\",    max_tokens: int = 8192,) -> dict:    \"\"\"    Validate total token count before sending to the model.    Raise early instead of silently truncating.    \"\"\"    enc = tiktoken.encoding_for_model(\"gpt-4\")  # approximation for GPT-5        total = (        len(enc.encode(system_prompt)) +        len(enc.encode(context)) +        len(enc.encode(question))    )        buffer = 1024  # reserve tokens for the response        return {        \"total_tokens\": total,        \"max_allowed\": max_tokens - buffer,        \"within_limit\": total <= (max_tokens - buffer),        \"overflow_by\": max(0, total - (max_tokens - buffer)),    }# Usage - call this before every LLM invocationresult = check_context_window(    system_prompt=ACTIVE_PROMPT.template,    context=format_docs(retrieved_nodes),    question=user_question,)if not result[\"within_limit\"]:    # Reduce top_k or truncate context, don't silently overflow    raise ValueError(        f\"Context window overflow: {result['overflow_by']} tokens over limit. \"        f\"Reduce similarity_top_k or truncate documents.\"    )\n```\n\nYour service gets a traffic spike. 500 concurrent requests hit your LLM API simultaneously. OpenAI returns 429 (rate limit exceeded). Your retry logic has no jitter all 500 requests retry at exactly the same interval. You hit the rate limit again. Repeat until the queue is exhausted or you’ve burned your error budget.\n\n``` python\nimport asyncioimport randomfrom openai import AsyncOpenAI, RateLimitErrorclient = AsyncOpenAI()async def call_llm_with_backoff(    messages: list,    model: str = \"gpt-5\",    max_retries: int = 4,) -> str:    \"\"\"    LLM call with exponential backoff + full jitter.    Prevents synchronized retry storms under rate limiting.    \"\"\"    for attempt in range(max_retries):        try:            response = await client.chat.completions.create(                model=model,                messages=messages,                temperature=0,                max_tokens=1024,            )            return response.choices[0].message.content        except RateLimitError:            if attempt == max_retries - 1:                raise            # Full jitter: random sleep up to cap            # Prevents all retries from firing at the same time            cap = 60  # max 60 second wait            sleep_time = random.uniform(0, min(cap, 1 * (2 ** attempt)))                        print(f\"Rate limited. Attempt {attempt + 1}/{max_retries}. \"                  f\"Waiting {sleep_time:.1f}s before retry.\")            await asyncio.sleep(sleep_time)        raise RuntimeError(\"Max retries exceeded on LLM API call\")\n```\n\nThis is where most teams get it completely wrong. They track HTTP status codes and latency, declare it “monitored,” and go home.\n\nThat’s not observability. That’s infrastructure monitoring.\n\nLLM observability requires an entirely different layer. The failure modes above hallucination, prompt drift, context truncation, degraded retrieval quality all look *fine* on your Datadog latency dashboard.\n\n``` python\nfrom prometheus_client import Counter, Histogram, Gaugeimport time# 1. Latency - time-to-first-token and total generation timeLLM_LATENCY = Histogram(    \"llm_request_duration_seconds\",    \"LLM API call duration\",    [\"model\", \"endpoint\"],    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0],)# 2. Token consumption - drives cost and context window awarenessTOKEN_USAGE = Counter(    \"llm_tokens_total\",    \"Total tokens consumed\",    [\"model\", \"token_type\"],  # token_type: input / output)# 3. Retrieval quality - are we fetching relevant context?RETRIEVAL_SCORE = Histogram(    \"retrieval_similarity_score\",    \"Cosine similarity of retrieved chunks to query\",    [\"index_name\"],    buckets=[0.5, 0.6, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0],)# 4. Answer quality - the metric most teams skipANSWER_CONFIDENCE = Gauge(    \"llm_answer_confidence\",    \"LLM judge score for answer quality (0-1)\",    [\"model\", \"query_type\"],)class InstrumentedLLMClient:    \"\"\"Wraps LLM calls with full observability instrumentation.\"\"\"    def __init__(self, model: str):        self.model = model        self.client = AsyncOpenAI()    async def chat(self, messages: list, query_type: str = \"general\") -> dict:        start = time.time()        response = await call_llm_with_backoff(messages, model=self.model)        duration = time.time() - start        # Record every call        LLM_LATENCY.labels(            model=self.model,            endpoint=\"chat\"        ).observe(duration)        return {\"response\": response, \"latency\": duration}The LLM judge: catching hallucinations automatically\n```\n\nStandard monitoring tells you nothing about answer quality. The only way to catch hallucinations at scale is to use a second LLM call as a judge, a model that evaluates whether the answer is actually grounded in the retrieved context.\n\n``` python\nasync def evaluate_answer_grounding(    question: str,    context: str,    answer: str,    judge_model: str = \"gpt-5\",  # use a capable model for judging) -> dict:    \"\"\"    LLM-as-judge: evaluates whether the answer is grounded in the context.    Returns a score 0-1 and a reason.    Score < 0.7 = likely hallucination. Alert on this.    \"\"\"    judge_prompt = f\"\"\"You are evaluating whether an AI answer is properly grounded in the provided context.Question: {question}Context provided to the AI:{context}AI Answer:{answer}Evaluate strictly:1. Does the answer use ONLY information from the context? (0.0 = uses outside knowledge, 1.0 = fully grounded)2. Is the answer factually consistent with the context? (0.0 = contradicts context, 1.0 = fully consistent)Respond with JSON only:{{\"grounding_score\": 0.0-1.0, \"consistency_score\": 0.0-1.0, \"reason\": \"brief explanation\"}}\"\"\"    response = await client.chat.completions.create(        model=judge_model,        messages=[{\"role\": \"user\", \"content\": judge_prompt}],        temperature=0,        response_format={\"type\": \"json_object\"},    )        import json    result = json.loads(response.choices[0].message.content)        # Record to Prometheus    overall_score = (result[\"grounding_score\"] + result[\"consistency_score\"]) / 2    ANSWER_CONFIDENCE.labels(        model=judge_model,        query_type=\"rag_grounding\"    ).set(overall_score)        # Alert threshold: below 0.7 = probable hallucination    if overall_score < 0.7:        print(f\"⚠️  Low grounding score: {overall_score:.2f}. \"              f\"Reason: {result['reason']}\")        return result\n```\n\nThe cost of LLM judges: Yes, you’re calling the LLM twice per request. At Gemini Flash prices ($0.30/1M), judging 10,000 answers/day with a 500-token judge prompt costs about $1.50/day. That’s cheaper than one customer support ticket for a hallucinated answer.\n\nDon’t judge every request in production. Sample 5–10%. That’s enough to catch systematic failures before they become incidents.\n\nHere’s what the full system looks like:\n\n```\nUser Request     │     ▼┌─────────────────┐│   FastAPI App   │ ← Rate limiting, auth, request validation└────────┬────────┘         │         ▼┌─────────────────┐│ Token Budget    │ ← Check context window before calling LLM│ Check           │└────────┬────────┘         │         ▼┌─────────────────┐     ┌──────────────────┐│  LlamaIndex     │────▶│   pgvector       ││  Retriever      │     │  (embeddings)    │└────────┬────────┘     └──────────────────┘         │         ▼┌─────────────────┐│  LangChain      │ ← Versioned prompt + model call with backoff│  Chain (LLM)    │└────────┬────────┘         │         ▼┌─────────────────┐│  LLM Judge      │ ← Sample 5-10% of responses for grounding check│  (async)        │└────────┬────────┘         │         ▼┌─────────────────┐│  Prometheus     │ ← Latency, tokens, retrieval score, answer quality│  + Grafana      │└─────────────────┘\n```\n\nWhen your LLM application pages you at 3AM, your first 10 minutes look different from any other service.\n\n**Standard service failure: check error rates → check recent deploys → rollback.**\n\nLLM application failure: the alert might not be an error rate. It’s a quality degradation that your users noticed before your monitoring did.\n\n```\n# Step 1: Is this a model provider outage?# Check status pages before doing anything else# https://status.openai.com# https://status.anthropic.com# https://status.cloud.google.com# Step 2: Check your LLM-specific metrics# If you wired up the Prometheus metrics above, you can query:# - llm_request_duration_seconds P99 (is latency spiking?)# - llm_tokens_total rate (are requests going through?)# - llm_answer_confidence (has quality dropped suddenly?)# - retrieval_similarity_score (is retrieval degraded?)# Step 3: Sample recent prompt/response pairs# Every LLM call should be logged with: prompt, context, response, latency, tokens# Pull the last 20 from the past 30 minutes and read them manually# This takes 5 minutes and catches what no metric can# Step 4: Check for prompt driftgit log --since=\"48 hours ago\" -- prompts.py  # did the prompt change?git diff HEAD~1 prompts.py                    # what changed?# Step 5: Check vector store for stale embeddings# If a major document update shipped recently, stale embeddings may be serving bad contextSELECT COUNT(*), MIN(created_at), MAX(created_at)FROM document_embeddingsWHERE created_at < NOW() - INTERVAL '7 days';\n```\n\nEverything above works with OpenAI, Anthropic, or Google APIs.\n\nIf you’re self-hosting Llama 3.3, Mistral Large 2, or DeepSeek V4, the architecture is the same the operational surface expands significantly.\n\nSelf-hosting only makes financial sense above roughly $50,000/month in API costs. Below that, the GPU infrastructure, the model serving maintenance (vLLM or TGI), the reliability engineering, and the upgrade cycle cost more than the API savings.\n\nIf you do go self-hosted:\n\n``` python\n# Replace the OpenAI client with an Ollama or vLLM endpointfrom openai import AsyncOpenAI# vLLM serves an OpenAI-compatible API# Just change the base_url - everything else stays the sameclient = AsyncOpenAI(    base_url=\"http://your-vllm-server:8000/v1\",    api_key=\"not-required-for-local\",)# Same call, different endpointresponse = await client.chat.completions.create(    model=\"meta-llama/Llama-3.3-70B-Instruct\",  # your deployed model    messages=[{\"role\": \"user\", \"content\": \"Hello\"}],)\n```\n\nThe observability layer stays identical. The failure modes change slightly — you now own model loading failures, GPU OOM errors, and model version management on top of everything above.\n\nIf you found this article helpful, a **clap** would help [Towards AI](https://pub.towardsai.net/?source=your_stories_outbox---writer_outbox_drafts-----------------------------------------) share it with more engineers building in this space.\n\nThe next article covers **LLM observability in depth**: exactly what to put in Grafana dashboards so you can tell the difference between a model provider hiccup and actual quality degradation.\n\nIf you’re earlier in the journey and building your first RAG pipeline, start with** ****Building a RAG Pipeline That Doesn’t Fall Apart**** first.**\n\n[Why Your LLM App Will Fail at 3AM (And How to Build One That Won’t)](https://pub.towardsai.net/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont-254d3940a968) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont", "canonical_source": "https://pub.towardsai.net/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont-254d3940a968?source=rss----98111c9905da---4", "published_at": "2026-07-14 19:31:00+00:00", "updated_at": "2026-07-14 19:53:03.891720+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-products", "ai-tools", "developer-tools"], "entities": ["Datadog", "New Relic", "Prometheus", "Gemini 2.5 Pro", "GPT-5.5", "Claude Opus 4.8", "LangChain", "LlamaIndex"], "alternates": {"html": "https://wpnews.pro/news/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont", "markdown": "https://wpnews.pro/news/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont.md", "text": "https://wpnews.pro/news/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont.txt", "jsonld": "https://wpnews.pro/news/why-your-llm-app-will-fail-at-3am-and-how-to-build-one-that-wont.jsonld"}}