{"slug": "your-openai-compatible-endpoint-hangs-for-180-seconds-then-fails-here-is-why", "title": "Your OpenAI-compatible endpoint hangs for 180 seconds, then fails. Here is why.", "summary": "A developer outlined a debugging approach for OpenAI-compatible LLM endpoints that stall for roughly three minutes before failing, arguing that the duration and a generic connection error do not identify the failing component. The writeup distinguishes connect, read, write and pool timeouts, notes that nginx's proxy_read_timeout measures gaps between upstream reads rather than the whole response, and provides a Python 3.11+ HTTPX example that retries only on specific status codes within a bounded overall deadline. It warns that settings such as streamIdleTimeoutMs and retryPolicy are application-specific and that string-matching exceptions for \"502\" or \"timeout\" can misclassify failures.", "body_md": "An LLM request that repeatedly stalls for roughly three minutes is a useful debugging clue, not a diagnosis. A client deadline, gateway timeout, upstream delay, buffering, or retries can all contribute. Neither a 178–194 second duration nor a message such as `TRANSPORT: Connection error` identifies the failing component on its own.\n\nStreaming HTTP chat completions typically deliver server-sent events (SSE). WebSocket support and automatic transport selection depend on the particular API, client, and version. There is no universal OpenAI-compatible SDK setting named `transport: sse`.\n\nKeys such as `api: openai-completions`, `streamIdleTimeoutMs`, and `retryPolicy` belong to particular applications or wrappers; copying them into an unrelated SDK may do nothing or raise a configuration error. Use your client's documented options and inspect its request logs. If a wrapper really does negotiate WebSockets first, test its documented HTTP/SSE option rather than treating that behavior as common to every SDK.\n\nThe official Python SDK documents streaming, timeout controls, and retry behavior separately in its [README](https://github.com/openai/openai-python). Check the documentation for the version you installed, including whether it already retries requests, before adding another retry layer.\n\nA connection timeout limits establishment of a connection. A read timeout limits a wait for incoming data; it is not necessarily a deadline for the whole response. A stream that keeps sending bytes can outlive a read timeout. A correctly implemented read timeout can also fire before the first body chunk, even if headers have arrived.\n\nHTTPX distinguishes connect, read, write, and pool timeouts in its [timeout documentation](https://www.python-httpx.org/advanced/timeouts/). Choose limits for the expected workload: a shorter timeout reduces waiting but may also interrupt a valid long-running generation. A fixed 90-to-180-second ratio is not generally appropriate.\n\nA proxy can delay forwarding upstream chunks. For an nginx HTTP proxy location serving SSE, this is a starting point, not a complete server configuration:\n\n```\n# Inside the existing location that proxies to the model backend:\nproxy_buffering off;\nproxy_cache off;\nproxy_read_timeout 90s;  # example: maximum gap between upstream reads\n```\n\n`proxy_read_timeout` is measured between successive reads, not across the entire response. nginx can also honor an upstream `X-Accel-Buffering: no` response header unless configured to ignore it. Other proxies, CDNs, compression layers, or the application itself may still buffer. Changing nginx alone does not guarantee token-by-token delivery. See the [nginx proxy module documentation](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering).\n\nString matching an exception for `\"502\"` or `\"timeout\"` can misclassify failures. Inspect transport exceptions and HTTP status codes. Repeating an unchanged malformed request or invalid credential is generally not useful; some authentication failures require refreshing credentials, not blindly resending them.\n\nThe following Python 3.11+ example uses HTTPX directly, not SDK-specific configuration. It demonstrates a non-streaming request so that the entire body is consumed inside the deadline. Configure `BASE`, `KEY`, and `MODEL` in your environment; `BASE` is the API base including `/v1` when required.\n\n``` python\nimport asyncio\nimport os\nimport random\nimport httpx\n\nRETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}\n\nasync def ask_with_retry(client, *, attempts=3, budget_s=60):\n    if attempts < 1 or budget_s <= 0:\n        raise ValueError(\"attempts and budget_s must be positive\")\n    # Includes attempts, response reads, and backoff.\n    async with asyncio.timeout(budget_s):\n        for attempt in range(attempts):\n            try:\n                response = await client.post(\n                    os.environ[\"BASE\"].rstrip(\"/\") + \"/chat/completions\",\n                    headers={\"Authorization\": f\"Bearer {os.environ['KEY']}\"},\n                    json={\n                        \"model\": os.environ[\"MODEL\"],\n                        \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}],\n                        \"stream\": False,\n                    },\n                )\n                response.raise_for_status()\n                return response.json()\n            except httpx.HTTPStatusError as exc:\n                if exc.response.status_code not in RETRYABLE_STATUS:\n                    raise\n                if attempt == attempts - 1:\n                    raise\n            except (httpx.TimeoutException, httpx.NetworkError):\n                if attempt == attempts - 1:\n                    raise\n            await asyncio.sleep(min(0.8 * 2 ** attempt, 6) + random.uniform(0, 0.3))\n\nasync def main():\n    async with httpx.AsyncClient(\n        timeout=httpx.Timeout(30, connect=5, read=20)\n    ) as client:\n        print(await ask_with_retry(client))\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\nThe status list and delays are example policy, not instructions to retry every server error. Production handling should respect a valid `Retry-After` within the remaining budget. A timed-out POST may already have reached the upstream, so repeating it can duplicate work or charges. Do not blindly replay a stream after delivering partial output or executing tool side effects.\n\nFor streaming, keep both the request and the full stream-consumption loop inside the deadline and close the stream on cancellation. Async timeouts use cooperative cancellation, not a process-level kill switch; blocking code or suppressed cancellation can delay cleanup. See [Python's timeout documentation](https://docs.python.org/3/library/asyncio-task.html#asyncio.timeout).\n\nThis shell example writes response headers and body to local files so an HTTP error is not mistaken for a successful generation. Set the three environment variables first and inspect the captured body without sharing credentials or sensitive content.\n\n```\ncurl -sS -N --connect-timeout 10 --max-time 60 \\\n  -D response.headers -o response.body \\\n  -w 'http=%{http_code} dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\\n' \\\n  \"$BASE/chat/completions\" \\\n  -H \"Authorization: Bearer $KEY\" \\\n  -H 'Content-Type: application/json' \\\n  -d \"$(python -c 'import json,os; print(json.dumps({\"model\":os.environ[\"MODEL\"],\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"stream\":True}))')\"\n```\n\n`time_connect` includes time from the start until connection establishment; DNS and TLS have their own timing fields. `time_starttransfer` measures the first response byte, often a header, not the first generated token. These are cumulative timings, not independent phase durations. `-N` disables curl's output buffering, not a proxy's. See the [curl manual](https://curl.se/docs/manpage.html).\n\nA fast test only shows that one request completed; it does not prove the client is at fault. Record HTTP status, headers, body/error events, curl's exit status, and arrival times of actual SSE events. Compare direct and proxied requests where available. A model-list endpoint can help, but presence in a list alone does not establish that the model is callable with your credentials.\n\nRelated guide: [Your OpenAI-compatible endpoint hangs for 180 seconds](https://csvtriage.com/g/openai-api-hangs-then-fails.html).\n\n**If you want the whole set:** I packaged all 30 of these recipes — pacing, JSON repair, failover ladders, evals, cost accounting — into one markdown bundle. It's $19, paid in USDT, download unlocks the moment the transfer confirms, no account needed.", "url": "https://wpnews.pro/news/your-openai-compatible-endpoint-hangs-for-180-seconds-then-fails-here-is-why", "canonical_source": "https://dev.to/csvtriage/your-openai-compatible-endpoint-hangs-for-180-seconds-then-fails-here-is-why-m7g", "published_at": "2026-11-06 23:40:22+00:00", "updated_at": "2026-11-06 23:53:27.929793+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["OpenAI", "HTTPX", "nginx", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-openai-compatible-endpoint-hangs-for-180-seconds-then-fails-here-is-why", "markdown": "https://wpnews.pro/news/your-openai-compatible-endpoint-hangs-for-180-seconds-then-fails-here-is-why.md", "text": "https://wpnews.pro/news/your-openai-compatible-endpoint-hangs-for-180-seconds-then-fails-here-is-why.txt", "jsonld": "https://wpnews.pro/news/your-openai-compatible-endpoint-hangs-for-180-seconds-then-fails-here-is-why.jsonld"}}