{"slug": "notes-from-a-2-am-on-call-when-llama-swap-s-model-reload-hit-a-30s-timeout-wall", "title": "Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall", "summary": "An engineer running a production LLM gateway traced recurring 502 errors during peak APAC hours to llama-swap model swaps that took 25-35 seconds on an A10G serving a 34B vLLM model, exceeding nginx's proxy_read_timeout before headers were sent. The fix combined raising proxy_read_timeout and proxy_send_timeout to 90s, disabling proxy_buffering, and adding a warmup endpoint that preloads models with a throwaway one-token completion during low-traffic windows. The engineer recommends setting timeout margins at least 2x the slowest measured swap time, or running two llama-swap instances behind a load balancer that routes to whichever already has the model warm.", "body_md": "\n\n``` php\n2026-10-17 02:14:33 ERROR: POST /v1/chat/completions -> 502 Bad Gateway\n2026-10-17 02:14:33 upstream prematurely closed connection while reading response header from upstream\n2026-10-17 02:14:48 llama-swap: model switch deepseek-coder-v2 -> qwen2.5-coder completed (29.4s)\n```\n\nOur production gateway started throwing 502s during peak APAC hours. The pattern was surgical: every request that triggered a model swap via `mostlygeek/llama-swap` died with `upstream prematurely closed connection`. The swap itself succeeded, but nginx had already killed the client socket.\n\nThe llama-swap README shows clean examples with instant model switching. What it doesn't tell you: when your vLLM backend is serving a 34B parameter model with `--max-model-len 16384` and `--gpu-memory-utilization 0.95`, unloading the KV cache + loading the new model weights takes 25-35 seconds on an A10G instance.\n\nStandard nginx tutorials suggest `proxy_read_timeout 60s` is generous. It's not. The timeout clock starts at request dispatch, not at first response byte. If llama-swap spends 29s swapping models before sending HTTP headers, you're already in the danger zone for concurrent requests piling up behind nginx's default `proxy_connect_timeout 60s`.\n\nThe issue wasn't llama-swap's logic—it was the interaction between:\n\n`proxy_read_timeout` starting before llama-swap could send a 202 Accepted`/v1/chat/completions` through a single llama-swap instance\nWhen testing the same setup against B-Lost's relay gateway (`https://b-lost.com/v1`) as a baseline comparison, their router returned headers within 850ms even during provider failover—they clearly batch-queue requests or use async model warming. We needed similar behavior.\n\n**The 3-line nginx fix:**\n\n```\nlocation /v1/ {\n    proxy_pass http://llama-swap:8000;\n    proxy_read_timeout 90s;        # raised from 60s\n    proxy_connect_timeout 10s;     # keep tight for fast-fail\n    proxy_send_timeout 90s;        # match read timeout\n    proxy_buffering off;           # critical: stream headers immediately\n}\n```\n\n**The llama-swap optimization:**\n\nWe added a dedicated warmup endpoint to pre-load models during low-traffic windows:\n\n``` python\nimport httpx\nimport asyncio\n\nasync def preload_model(model: str):\n    async with httpx.AsyncClient(timeout=120.0) as client:\n        # Trigger swap with a throwaway completion\n        await client.post(\n            \"http://llama-swap:8000/v1/chat/completions\",\n            json={\n                \"model\": model,\n                \"messages\": [{\"role\": \"user\", \"content\": \"ping\"}],\n                \"max_tokens\": 1\n            }\n        )\n\n# Run during deployment or via cron before peak hours\nawait preload_model(\"qwen2.5-coder:32b\")\n```\n\n**Latency impact:**\n\n**Never deploy llama-swap without load-tested timeout margins that exceed your slowest model's swap duration by 2x.** \n\nMeasure swap time empirically:\n\n```\ntime curl -X POST http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"new-model\",\"messages\":[{\"role\":\"user\",\"content\":\"test\"}],\"max_tokens\":1}'\n```\n\nIf the response takes 28s, your `proxy_read_timeout` must be ≥56s. If you can't afford that client-perceived latency, you need model preloading automation—not wishful thinking about sub-second swaps.\n\nThe production-grade move: run two llama-swap instances with a smart load balancer that routes to whichever instance already has the requested model warm. That's the architecture B-Lost and other high-reliability gateways use under the hood. For our scale, the preload script plus nginx tuning was enough to stop the 2 AM pages.", "url": "https://wpnews.pro/news/notes-from-a-2-am-on-call-when-llama-swap-s-model-reload-hit-a-30s-timeout-wall", "canonical_source": "https://dev.to/dave_mille/notes-from-a-2-am-on-call-when-llama-swaps-model-reload-hit-a-30s-timeout-wall-38mp", "published_at": "2026-10-18 15:45:40+00:00", "updated_at": "2026-10-18 15:47:28.939799+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["llama-swap", "vLLM", "nginx", "B-Lost", "qwen2.5-coder", "deepseek-coder-v2", "httpx"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/notes-from-a-2-am-on-call-when-llama-swap-s-model-reload-hit-a-30s-timeout-wall", "markdown": "https://wpnews.pro/news/notes-from-a-2-am-on-call-when-llama-swap-s-model-reload-hit-a-30s-timeout-wall.md", "text": "https://wpnews.pro/news/notes-from-a-2-am-on-call-when-llama-swap-s-model-reload-hit-a-30s-timeout-wall.txt", "jsonld": "https://wpnews.pro/news/notes-from-a-2-am-on-call-when-llama-swap-s-model-reload-hit-a-30s-timeout-wall.jsonld"}}