From Prototype to Production: Deploying an LLM App That Won't Collapse

An engineer details the architecture and engineering discipline required to move an LLM application from prototype to production, addressing scaling, rate limits, and failure modes. The piece uses a case study of an e-commerce startup in Dubai whose search-and-summarize assistant collapsed under load, and outlines options for model hosting, stateless API design, and queueing strategies.

A field guide to turning a working notebook into an application that survives real traffic — serving, scaling, rate limits, backpressure, and the failure modes nobody demos. Three weeks before launch, the founder of an e-commerce startup in Dubai called me with a tone I now recognize as controlled panic. His team had built a search-and-summarize assistant over a catalog of 12,000 products. The demo was beautiful. It ran on a laptop, answered in about a second, and he was two days from showing it to investors. Then he load-tested it with 40 simulated users. Response time went from 1.2 seconds to 47 seconds. Memory on the model server climbed past 30 GB, and the API began returning HTTP 504 errors. Requests queued without any bound, and when a single worker fell over, it took the whole process down with it. "Just add more servers, right?" he asked. I have had some version of this conversation two dozen times in the last two years. The honest answer is: no. Throwing hardware at a prototype that was never built to be an application is how you burn a six-figure cloud bill and still ship a service that collapses at the worst possible moment. The gap between "my model works" and "my product does not fall over" is a specific, learnable engineering discipline. This article is what that discipline looks like — the architecture, the code, the numbers, and the failure modes I learned the expensive way. The fastest way to diagnose the gap is to ask one question: what happens when 40 people use it at the same time? A prototype answers with a crash. An application answers with a designed response — a queue, a rate limit, a fallback, or a scaled-up worker. Four properties separate the two, and everything in this article hangs off them: The e-commerce team had none of the four. Their model, the FastAPI app, and the session cache were one process. When inference slowed, every request blocked on the same code path, and memory grew with every queued conversation. Fixing it was not a hardware problem. It was an architecture problem. Before anything else, decide where the model runs. There are three honest options, and your choice determines almost everything downstream. | Option | Cost per 1M tokens | Latency | Operational load | |---|---|---|---| | Hosted API OpenAI-compatible gateway | $2–$15 | 300–900 ms | Near zero | | Self-hosted GPU vLLM / TGI / TensorRT-LLM | $0.5–$4 on an 8x H100 box | 40–150 ms | High: drivers, VRAM, autoscaling | | Quantized CPU llama.cpp / Ollama | ~$0.05–$0.30 | 2–8 s | Medium, surprisingly robust | My default in 2026 is: a hosted endpoint for the main model, self-hosted only for high-volume or data-sensitive workloads. The startup in Dubai handled payment and inventory data that could not leave their VPC, so they went self-hosted with vLLM. If you do not have that constraint, start with a hosted endpoint and postpone infrastructure until you have users to justify it. The one thing I insist on regardless: put the model behind a single interface so you can swap hosted for self-hosted with an environment variable. Build that abstraction in week one, because you will flip it at some point, and you do not want to be rewriting call sites during an outage. Here is the layout I ship, and it is boring on purpose: Client ─▶ Nginx TLS, rate limits ─▶ API replicas FastAPI, stateless │ ├─▶ Redis sessions, cache, queue ├─▶ Postgres facts, audit logs └─▶ vLLM GPU / hosted LLM API Three rules hold it together. First, the API layer is stateless — every conversation ID maps to a Redis key, never to process memory, so you can run 20 replicas and kill any of them mid-flight. Second, the model is behind the same contract whether it is a local vLLM server or a hosted endpoint. Third, everything the model needs is assembled at request time: the context window is built from Redis and Postgres, used, and discarded. Your notebook code will not survive this. Here is the shape of the service layer, with the two details most people skip: an explicit input budget and a hard timeout on the model call. python from fastapi import FastAPI, HTTPException from pydantic import BaseModel, Field app = FastAPI class ChatRequest BaseModel : conversation id: str message: str = Field ..., max length=4 000 @app.post "/v1/chat" async def chat req: ChatRequest, user=Depends require auth : 1. Auth: validate the caller API key / JWT . Never skip this. 2. Load the last 10 turns from Redis bounded window . turns = await redis.lrange f"conv:{req.conversation id}", 0, 9 3. Call the model with a hard timeout. try: response = await asyncio.wait for model.complete turns + {"role": "user", "content": req.message} , timeout=10.0, except asyncio.TimeoutError: raise HTTPException 503, "model busy — retry shortly" 4. Persist the turns and return. await redis.rpush f"conv:{req.conversation id}", req.message, response return {"reply": response} The asyncio.wait for line is the single most important line in this file. Without it, a slow model backs up every worker, the queue grows without bound, and the 504s the startup saw are guaranteed. With it, the failure is contained: one request fails with a clear message and the rest of the system stays alive. I cannot overstate how often the entire "it collapsed under load" story reduces to a missing timeout. For the model abstraction: python class LLMBackend Protocol : async def complete self, messages: list dict , stream: bool = False - str: ... class HostedBackend: OpenAI-compatible endpoint async def complete self, messages, stream=False : return await client.chat.completions.create model="your-model", messages=messages, stream=stream, timeout=15.0 class LocalBackend: vLLM / TGI on your GPU box async def complete self, messages, stream=False : return await httpx.AsyncClient timeout=30.0 .post "http://llm-server:8000/v1/chat/completions", json={"model": "self-hosted", "messages": messages} Swap the two with one setting in your config. The interface is the whole point — it is what lets you migrate without an outage. Run the API behind a real ASGI server — never uvicorn's dev mode in front of users. For FastAPI that means a process manager running multiple workers: gunicorn app.main:app \ --worker-class uvicorn.workers.UvicornWorker \ --workers 4 --threads 8 \ --timeout 60 --graceful-timeout 30 The rule I use: workers = 2 × vCPUs, a hard timeout that matches your worst accepted latency, and Nginx or Traefik in front for TLS and connection limits. Then autoscale at the right layer: the model server scales on GPU utilization and queue depth, not CPU — a saturated GPU looks idle to the CPU metric, and autoscaling on the wrong signal is how you get paged at 3 AM. If you self-host the model, do not hand-roll an inference loop. vLLM, TGI, or TensorRT-LLM give you continuous batching, which is the difference between a GPU running at 15% utilization and one running at 85%. Continuous batching means the server never waits for the full batch to finish — it evicts completed sequences mid-run and packs in new ones. On an 8x H100 box, moving a summarization workload from a naive loop to vLLM took throughput from 180 requests/minute to 780 requests/minute on the same hardware. That is not a tuning win. It is a different product. Every endpoint needs limits, and the LLM endpoint needs them most, because a burst of traffic there converts directly into model spend. I enforce two layers. First, the edge: limit req zone $binary remote addr zone=llm:10m rate=20r/m; location /v1/chat { limit req zone=llm burst=5 nodelay; proxy pass http://api; } Second, an application-level token budget per conversation, because a runaway agent loop can burn more tokens in an hour than ten thousand real users. I had a client whose support agent hit a retry bug and spent about $180 in a single night on a self-hosted box plus a hosted fallback. The rate limiter protected the server; the token budget protected the bank account. Add both: spent = await redis.get f"budget:{req.conversation id}" or 0 if int spent 50 000: ~50k tokens per conversation raise HTTPException 429, "conversation budget exhausted" ... after the model call ... await redis.incrby f"budget:{req.conversation id}", tokens used For long answers, stream tokens out as they are generated. Streaming turns an 8-second full answer into a first-token delay under a second, and users perceive it as dramatically faster. Use Server-Sent Events, pass stream=True through the model client, and flush each chunk to the client as it arrives. The API shape barely changes and perceived latency collapses. The fastest inference is the one you do not run. Cache exact-match prompts and, more importantly, cache retrieval and tool outputs — the same document chunks get embedded over and over. A Redis cache with a one-hour TTL on retrieval chunks cut the e-commerce app's upstream model calls by roughly 60%. Also cache the big static pieces: system prompts, schema definitions, and any pre-computed context that does not change per user. Every cached token is money you do not spend. Add this in week one, not when it breaks. With OpenTelemetry + Prometheus, track at minimum: A dashboard that shows all five has saved me more than once. A friend's inference service took ten days to notice that p99 had silently tripled — queue depth grew a little every day and nobody was watching it. Here is the part nobody puts in the demo: Self-hosting is the sexiest option and often the wrong one. Do not run your own model server if your traffic is below steady state, if you have nobody on call who understands GPU driver updates, or if your data can legally live with a hosted provider. A hosted API at $10 per million tokens will be cheaper than a $1,800-a-month GPU box sitting at 8% utilization because you have 200 users. Self-host when the math, the data, or the latency requirement forces it — not because it feels like "real" AI engineering. When you ship an LLM app, walk this list before you call it done: We spent nine days rebuilding. The model moved behind vLLM with continuous batching, sessions went to Redis, gunicorn ran four stateless workers behind Nginx with rate limits, and every model call got a timeout. Forty concurrent users stopped being a disaster. At launch it handled 400 concurrent users — median 1.4 seconds, p95 4.1 seconds — on a bill a third of the original estimate, because nothing was over-provisioned to mask a broken design. The demo worked exactly the same. The difference was that now, when something failed, it failed in one bounded place, loudly, with a readable log — instead of taking the entire product down. That is the whole point of this discipline. A production LLM app is not a model that performs well. It is a system that fails well. Build for the failure, and the success takes care of itself. If you are deploying your first LLM app, start with the checklist and the timeout. The rest is iteration. Gulshan Yad