Your OpenAI-compatible endpoint hangs for 180 seconds, then fails. Here is why.

A developer outlined a debugging approach for OpenAI-compatible LLM endpoints that stall for roughly three minutes before failing, arguing that the duration and a generic connection error do not identify the failing component. The writeup distinguishes connect, read, write and pool timeouts, notes that nginx's proxy_read_timeout measures gaps between upstream reads rather than the whole response, and provides a Python 3.11+ HTTPX example that retries only on specific status codes within a bounded overall deadline. It warns that settings such as streamIdleTimeoutMs and retryPolicy are application-specific and that string-matching exceptions for "502" or "timeout" can misclassify failures.

An LLM request that repeatedly stalls for roughly three minutes is a useful debugging clue, not a diagnosis. A client deadline, gateway timeout, upstream delay, buffering, or retries can all contribute. Neither a 178–194 second duration nor a message such as TRANSPORT: Connection error identifies the failing component on its own. Streaming HTTP chat completions typically deliver server-sent events SSE . WebSocket support and automatic transport selection depend on the particular API, client, and version. There is no universal OpenAI-compatible SDK setting named transport: sse . Keys such as api: openai-completions , streamIdleTimeoutMs , and retryPolicy belong to particular applications or wrappers; copying them into an unrelated SDK may do nothing or raise a configuration error. Use your client's documented options and inspect its request logs. If a wrapper really does negotiate WebSockets first, test its documented HTTP/SSE option rather than treating that behavior as common to every SDK. The official Python SDK documents streaming, timeout controls, and retry behavior separately in its README https://github.com/openai/openai-python . Check the documentation for the version you installed, including whether it already retries requests, before adding another retry layer. A connection timeout limits establishment of a connection. A read timeout limits a wait for incoming data; it is not necessarily a deadline for the whole response. A stream that keeps sending bytes can outlive a read timeout. A correctly implemented read timeout can also fire before the first body chunk, even if headers have arrived. HTTPX distinguishes connect, read, write, and pool timeouts in its timeout documentation https://www.python-httpx.org/advanced/timeouts/ . Choose limits for the expected workload: a shorter timeout reduces waiting but may also interrupt a valid long-running generation. A fixed 90-to-180-second ratio is not generally appropriate. A proxy can delay forwarding upstream chunks. For an nginx HTTP proxy location serving SSE, this is a starting point, not a complete server configuration: Inside the existing location that proxies to the model backend: proxy buffering off; proxy cache off; proxy read timeout 90s; example: maximum gap between upstream reads proxy read timeout is measured between successive reads, not across the entire response. nginx can also honor an upstream X-Accel-Buffering: no response header unless configured to ignore it. Other proxies, CDNs, compression layers, or the application itself may still buffer. Changing nginx alone does not guarantee token-by-token delivery. See the nginx proxy module documentation https://nginx.org/en/docs/http/ngx http proxy module.html proxy buffering . String matching an exception for "502" or "timeout" can misclassify failures. Inspect transport exceptions and HTTP status codes. Repeating an unchanged malformed request or invalid credential is generally not useful; some authentication failures require refreshing credentials, not blindly resending them. The following Python 3.11+ example uses HTTPX directly, not SDK-specific configuration. It demonstrates a non-streaming request so that the entire body is consumed inside the deadline. Configure BASE , KEY , and MODEL in your environment; BASE is the API base including /v1 when required. python import asyncio import os import random import httpx RETRYABLE STATUS = {408, 429, 500, 502, 503, 504} async def ask with retry client, , attempts=3, budget s=60 : if attempts < 1 or budget s <= 0: raise ValueError "attempts and budget s must be positive" Includes attempts, response reads, and backoff. async with asyncio.timeout budget s : for attempt in range attempts : try: response = await client.post os.environ "BASE" .rstrip "/" + "/chat/completions", headers={"Authorization": f"Bearer {os.environ 'KEY' }"}, json={ "model": os.environ "MODEL" , "messages": {"role": "user", "content": "hi"} , "stream": False, }, response.raise for status return response.json except httpx.HTTPStatusError as exc: if exc.response.status code not in RETRYABLE STATUS: raise if attempt == attempts - 1: raise except httpx.TimeoutException, httpx.NetworkError : if attempt == attempts - 1: raise await asyncio.sleep min 0.8 2 attempt, 6 + random.uniform 0, 0.3 async def main : async with httpx.AsyncClient timeout=httpx.Timeout 30, connect=5, read=20 as client: print await ask with retry client if name == " main ": asyncio.run main The status list and delays are example policy, not instructions to retry every server error. Production handling should respect a valid Retry-After within the remaining budget. A timed-out POST may already have reached the upstream, so repeating it can duplicate work or charges. Do not blindly replay a stream after delivering partial output or executing tool side effects. For streaming, keep both the request and the full stream-consumption loop inside the deadline and close the stream on cancellation. Async timeouts use cooperative cancellation, not a process-level kill switch; blocking code or suppressed cancellation can delay cleanup. See Python's timeout documentation https://docs.python.org/3/library/asyncio-task.html asyncio.timeout . This shell example writes response headers and body to local files so an HTTP error is not mistaken for a successful generation. Set the three environment variables first and inspect the captured body without sharing credentials or sensitive content. curl -sS -N --connect-timeout 10 --max-time 60 \ -D response.headers -o response.body \ -w 'http=%{http code} dns=%{time namelookup}s connect=%{time connect}s tls=%{time appconnect}s ttfb=%{time starttransfer}s total=%{time total}s\n' \ "$BASE/chat/completions" \ -H "Authorization: Bearer $KEY" \ -H 'Content-Type: application/json' \ -d "$ python -c 'import json,os; print json.dumps {"model":os.environ "MODEL" ,"messages": {"role":"user","content":"hi"} ,"stream":True} ' " time connect includes time from the start until connection establishment; DNS and TLS have their own timing fields. time starttransfer measures the first response byte, often a header, not the first generated token. These are cumulative timings, not independent phase durations. -N disables curl's output buffering, not a proxy's. See the curl manual https://curl.se/docs/manpage.html . A fast test only shows that one request completed; it does not prove the client is at fault. Record HTTP status, headers, body/error events, curl's exit status, and arrival times of actual SSE events. Compare direct and proxied requests where available. A model-list endpoint can help, but presence in a list alone does not establish that the model is callable with your credentials. Related guide: Your OpenAI-compatible endpoint hangs for 180 seconds https://csvtriage.com/g/openai-api-hangs-then-fails.html . If you want the whole set: I packaged all 30 of these recipes — pacing, JSON repair, failover ladders, evals, cost accounting — into one markdown bundle. It's $19, paid in USDT, download unlocks the moment the transfer confirms, no account needed.