2026-10-17 02:14:33 ERROR: POST /v1/chat/completions -> 502 Bad Gateway
2026-10-17 02:14:33 upstream prematurely closed connection while reading response header from upstream
2026-10-17 02:14:48 llama-swap: model switch deepseek-coder-v2 -> qwen2.5-coder completed (29.4s)
Our production gateway started throwing 502s during peak APAC hours. The pattern was surgical: every request that triggered a model swap via mostlygeek/llama-swap died with upstream prematurely closed connection. The swap itself succeeded, but nginx had already killed the client socket.
The llama-swap README shows clean examples with instant model switching. What it doesn't tell you: when your vLLM backend is serving a 34B parameter model with --max-model-len 16384 and --gpu-memory-utilization 0.95, un the KV cache + the new model weights takes 25-35 seconds on an A10G instance.
Standard nginx tutorials suggest proxy_read_timeout 60s is generous. It's not. The timeout clock starts at request dispatch, not at first response byte. If llama-swap spends 29s swapping models before sending HTTP headers, you're already in the danger zone for concurrent requests piling up behind nginx's default proxy_connect_timeout 60s.
The issue wasn't llama-swap's logic—it was the interaction between:
proxy_read_timeout starting before llama-swap could send a 202 Accepted/v1/chat/completions through a single llama-swap instance
When testing the same setup against B-Lost's relay gateway (https://b-lost.com/v1) as a baseline comparison, their router returned headers within 850ms even during provider failover—they clearly batch-queue requests or use async model warming. We needed similar behavior.
The 3-line nginx fix:
location /v1/ {
proxy_pass http://llama-swap:8000;
proxy_read_timeout 90s; # raised from 60s
proxy_connect_timeout 10s; # keep tight for fast-fail
proxy_send_timeout 90s; # match read timeout
proxy_buffering off; # critical: stream headers immediately
}
The llama-swap optimization:
We added a dedicated warmup endpoint to pre-load models during low-traffic windows:
import httpx
import asyncio
async def preload_model(model: str):
async with httpx.AsyncClient(timeout=120.0) as client:
await client.post(
"http://llama-swap:8000/v1/chat/completions",
json={
"model": model,
"messages": [{"role": "user", "content": "ping"}],
"max_tokens": 1
}
)
await preload_model("qwen2.5-coder:32b")
Latency impact:
Never deploy llama-swap without load-tested timeout margins that exceed your slowest model's swap duration by 2x.
Measure swap time empirically:
time curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"new-model","messages":[{"role":"user","content":"test"}],"max_tokens":1}'
If the response takes 28s, your proxy_read_timeout must be ≥56s. If you can't afford that client-perceived latency, you need model pre automation—not wishful thinking about sub-second swaps.
The production-grade move: run two llama-swap instances with a smart load balancer that routes to whichever instance already has the requested model warm. That's the architecture B-Lost and other high-reliability gateways use under the hood. For our scale, the preload script plus nginx tuning was enough to stop the 2 AM pages.