From Prototype to Production: Deploying an LLM App That Won't Collapse

A field guide to turning a working notebook into an application that survives real traffic — serving, scaling, rate limits, backpressure, and the failure modes nobody demos.

Three weeks before launch, the founder of an e-commerce startup in Dubai called me with a tone I now recognize as controlled panic. His team had built a search-and-summarize assistant over a catalog of 12,000 products. The demo was beautiful. It ran on a laptop, answered in about a second, and he was two days from showing it to investors. Then he load-tested it with 40 simulated users. Response time went from 1.2 seconds to 47 seconds. Memory on the model server climbed past 30 GB, and the API began returning HTTP 504 errors. Requests queued without any bound, and when a single worker fell over, it took the whole process down with it.

"Just add more servers, right?" he asked.

I have had some version of this conversation two dozen times in the last two years. The honest answer is: no. Throwing hardware at a prototype that was never built to be an application is how you burn a six-figure cloud bill and still ship a service that collapses at the worst possible moment. The gap between "my model works" and "my product does not fall over" is a specific, learnable engineering discipline. This article is what that discipline looks like — the architecture, the code, the numbers, and the failure modes I learned the expensive way.

The fastest way to diagnose the gap is to ask one question: what happens when 40 people use it at the same time? A prototype answers with a crash. An application answers with a designed response — a queue, a rate limit, a fallback, or a scaled-up worker.

Four properties separate the two, and everything in this article hangs off them:

The e-commerce team had none of the four. Their model, the FastAPI app, and the session cache were one process. When inference slowed, every request blocked on the same code path, and memory grew with every queued conversation. Fixing it was not a hardware problem. It was an architecture problem.

Before anything else, decide where the model runs. There are three honest options, and your choice determines almost everything downstream.

My default in 2026 is: a hosted endpoint for the main model, self-hosted only for high-volume or data-sensitive workloads. The startup in Dubai handled payment and inventory data that could not leave their VPC, so they went self-hosted with vLLM. If you do not have that constraint, start with a hosted endpoint and postpone infrastructure until you have users to justify it.

The one thing I insist on regardless: put the model behind a single interface so you can swap hosted for self-hosted with an environment variable. Build that abstraction in week one, because you will flip it at some point, and you do not want to be rewriting call sites during an outage.

Here is the layout I ship, and it is boring on purpose:

Client ─▶ Nginx (TLS, rate limits) ─▶ API replicas (FastAPI, stateless)
                                         │
                                         ├─▶ Redis (sessions, cache, queue)
                                         ├─▶ Postgres (facts, audit logs)
                                         └─▶ vLLM (GPU) / hosted LLM API

Three rules hold it together. First, the API layer is stateless — every conversation ID maps to a Redis key, never to process memory, so you can run 20 replicas and kill any of them mid-flight. Second, the model is behind the same contract whether it is a local vLLM server or a hosted endpoint. Third, everything the model needs is assembled at request time: the context window is built from Redis and Postgres, used, and discarded.

Your notebook code will not survive this. Here is the shape of the service layer, with the two details most people skip: an explicit input budget and a hard timeout on the model call.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

app = FastAPI()

class ChatRequest(BaseModel):
    conversation_id: str
    message: str = Field(..., max_length=4_000)

@app.post("/v1/chat")
async def chat(req: ChatRequest, user=Depends(require_auth)):
    turns = await redis.lrange(f"conv:{req.conversation_id}", 0, 9)
    try:
        response = await asyncio.wait_for(
            model.complete(turns + [{"role": "user", "content": req.message}]),
            timeout=10.0,
        )
    except asyncio.TimeoutError:
        raise HTTPException(503, "model busy — retry shortly")
    await redis.rpush(f"conv:{req.conversation_id}", req.message, response)
    return {"reply": response}

The asyncio.wait_for

line is the single most important line in this file. Without it, a slow model backs up every worker, the queue grows without bound, and the 504s the startup saw are guaranteed. With it, the failure is contained: one request fails with a clear message and the rest of the system stays alive. I cannot overstate how often the entire "it collapsed under load" story reduces to a missing timeout.

For the model abstraction:

class LLMBackend(Protocol):
    async def complete(self, messages: list[dict], stream: bool = False) -> str: ...

class HostedBackend:   # OpenAI-compatible endpoint
    async def complete(self, messages, stream=False):
        return await client.chat.completions.create(
            model="your-model", messages=messages,
            stream=stream, timeout=15.0)

class LocalBackend:    # vLLM / TGI on your GPU box
    async def complete(self, messages, stream=False):
        return await httpx.AsyncClient(timeout=30.0).post(
            "http://llm-server:8000/v1/chat/completions",
            json={"model": "self-hosted", "messages": messages})

Swap the two with one setting in your config. The interface is the whole point — it is what lets you migrate without an outage.

Run the API behind a real ASGI server — never uvicorn's dev mode in front of users. For FastAPI that means a process manager running multiple workers:

gunicorn app.main:app \
  --worker-class uvicorn.workers.UvicornWorker \
  --workers 4 --threads 8 \
  --timeout 60 --graceful-timeout 30

The rule I use: workers = 2 × vCPUs, a hard timeout that matches your worst accepted latency, and Nginx or Traefik in front for TLS and connection limits. Then autoscale at the right layer: the model server scales on GPU utilization and queue depth, not CPU — a saturated GPU looks idle to the CPU metric, and autoscaling on the wrong signal is how you get paged at 3 AM.

If you self-host the model, do not hand-roll an inference loop. vLLM, TGI, or TensorRT-LLM give you continuous batching, which is the difference between a GPU running at 15% utilization and one running at 85%. Continuous batching means the server never waits for the full batch to finish — it evicts completed sequences mid-run and packs in new ones. On an 8x H100 box, moving a summarization workload from a naive loop to vLLM took throughput from 180 requests/minute to 780 requests/minute on the same hardware. That is not a tuning win. It is a different product.

Every endpoint needs limits, and the LLM endpoint needs them most, because a burst of traffic there converts directly into model spend. I enforce two layers. First, the edge:

limit_req_zone $binary_remote_addr zone=llm:10m rate=20r/m;
location /v1/chat {
    limit_req zone=llm burst=5 nodelay;
    proxy_pass http://api;
}

Second, an application-level token budget per conversation, because a runaway agent loop can burn more tokens in an hour than ten thousand real users. I had a client whose support agent hit a retry bug and spent about $180 in a single night on a self-hosted box plus a hosted fallback. The rate limiter protected the server; the token budget protected the bank account. Add both:

spent = await redis.get(f"budget:{req.conversation_id}") or 0
if int(spent) > 50_000:                     # ~50k tokens per conversation
    raise HTTPException(429, "conversation budget exhausted")
await redis.incrby(f"budget:{req.conversation_id}", tokens_used)

For long answers, stream tokens out as they are generated. Streaming turns an 8-second full answer into a first-token delay under a second, and users perceive it as dramatically faster. Use Server-Sent Events, pass stream=True

through the model client, and flush each chunk to the client as it arrives. The API shape barely changes and perceived latency collapses.

The fastest inference is the one you do not run. Cache exact-match prompts and, more importantly, cache retrieval and tool outputs — the same document chunks get embedded over and over. A Redis cache with a one-hour TTL on retrieval chunks cut the e-commerce app's upstream model calls by roughly 60%. Also cache the big static pieces: system prompts, schema definitions, and any pre-computed context that does not change per user. Every cached token is money you do not spend.

Add this in week one, not when it breaks. With OpenTelemetry + Prometheus, track at minimum:

A dashboard that shows all five has saved me more than once. A friend's inference service took ten days to notice that p99 had silently tripled — queue depth grew a little every day and nobody was watching it.

Here is the part nobody puts in the demo:

Self-hosting is the sexiest option and often the wrong one. Do not run your own model server if your traffic is below steady state, if you have nobody on call who understands GPU driver updates, or if your data can legally live with a hosted provider. A hosted API at $10 per million tokens will be cheaper than a $1,800-a-month GPU box sitting at 8% utilization because you have 200 users. Self-host when the math, the data, or the latency requirement forces it — not because it feels like "real" AI engineering.

When you ship an LLM app, walk this list before you call it done:

We spent nine days rebuilding. The model moved behind vLLM with continuous batching, sessions went to Redis, gunicorn ran four stateless workers behind Nginx with rate limits, and every model call got a timeout. Forty concurrent users stopped being a disaster. At launch it handled 400 concurrent users — median 1.4 seconds, p95 4.1 seconds — on a bill a third of the original estimate, because nothing was over-provisioned to mask a broken design.

The demo worked exactly the same. The difference was that now, when something failed, it failed in one bounded place, loudly, with a readable log — instead of taking the entire product down.

That is the whole point of this discipline. A production LLM app is not a model that performs well. It is a system that fails well. Build for the failure, and the success takes care of itself.

If you are deploying your first LLM app, start with the checklist and the timeout. The rest is iteration.

*Gulshan Yad