{"slug": "622-of-our-5087-llm-api-calls-never-returned-an-answer", "title": "622 of our 5,087 LLM API calls never returned an answer", "summary": "A production LLM gateway logged 5,087 chat completions between 28 June and 10 September 2026, of which 622 (12.2%) returned no answer to the caller, according to an analysis of the failure data. The largest share was 503 errors at 6.88%, indicating upstream provider capacity shortages, followed by 400 invalid-request errors at 2.40%, and the writeup argues retry logic should only retry 429 and 5xx responses while capping retry budgets in tokens rather than attempts.", "body_md": "Every tutorial about LLM pricing counts calls. You send a request, you get an answer, you multiply. The arithmetic is clean because it assumes something that isn't true: that a call returns an answer.\n\nBetween 28 June and 10 September 2026 our gateway logged 5,087 chat completions. 622 of them — 12.2% — ended with no answer for the caller. That is not an outage. That is the normal weather of a production LLM client, and nobody puts it in the cost model.\n\nTwo words before the numbers. A **token** is roughly ¾ of a word; you are billed per million, separately for **input** (what you send) and **output** (what the model writes back). And every failure arrives as an **HTTP status code** — a three-digit number where 4xx means \"your request was wrong\" and 5xx means \"something on our side broke\". That distinction turns out to be the whole article.\n\n| Outcome | Status | Share of all calls | \n|---|---|---|\n| no provider available | 503 | 6.88% | \n| invalid request | 400 | 2.40% | \n| client closed connection | 499 | 1.51% | \n| timeout | 504 | 0.94% | \n| out of credit | 402 | 0.28% | \n\nThe biggest slice isn't your code. It's 503 — the upstream vendor had no capacity for that model at that second. The second biggest *is* your code: a malformed request, an unsupported parameter, a role the endpoint doesn't accept.\n\nThat split matters because most retry wrappers don't make it. The default shape everyone copies — catch the exception, sleep, try again, three times — treats a 400 exactly like a 503. A 503 has a real chance of succeeding on the next attempt. A 400 will fail identically until the heat death of the universe. Of our 622 failures, 137 were terminal in that way: 400, 402, 404. Retried three times each, that is 411 requests that could never have produced anything.\n\nHere is the part that surprised me. A rejected request is cheap. A 400 never reaches the model, so no tokens are generated and nothing is billed. A 503 means the request never started. You lose latency, not dollars.\n\nTimeouts and cancellations are different. A 504 means the model *was* generating — you just stopped waiting. A 499 means your own user hit stop, or your HTTP client's deadline fired, after the answer had already started coming back. The tokens exist. Somebody generated them. In our logs that's 2.45% of all calls, and it is the only category where the meter was actually running.\n\nThen you retry, and the second attempt pays for the entire input again. Not the remainder — the whole prompt, from the system message down. This is why the number to track is **attempts per answer**, not calls. At `gpt-5.6`'s official $2.50 per million input tokens, our median prompt of 1,290 tokens costs about a third of a cent per attempt. Trivial, until a job with a 40,000-token context retries twice under load and you've bought that context three times for one answer.\n\nTwo lines of policy, and they're both about classification rather than volume.\n\n``` python\nRETRYABLE = {429, 500, 502, 503, 504}\n\ndef call_with_budget(send, payload, max_input_tokens=60_000):\n    spent = 0\n    for attempt in range(3):\n        try:\n            return send(payload)\n        except APIError as e:\n            spent += payload[\"approx_input_tokens\"]\n            if e.status not in RETRYABLE or spent > max_input_tokens:\n                raise\n            time.sleep(2 ** attempt)\n    raise RuntimeError(\"retry budget exhausted\")\n```\n\nFirst: retry only 429 and 5xx. Everything in the 4xx family except 429 is a bug report addressed to you, and sleeping on it won't fix your JSON. Second: cap the retry budget in **tokens**, not attempts. Three attempts on a 500-token prompt and three attempts on a 40,000-token prompt are the same line of code and an eighty-fold difference in what you bought. The budget is the thing that scales; the attempt count isn't.\n\nOne more, cheaper still: turn on **streaming**, where tokens arrive as they're produced instead of all at once at the end. A timeout on a streamed response leaves you holding a partial answer you can show or salvage. A timeout on a blocking call leaves you holding nothing, having paid the same.\n\nOur own gateway records zero tokens and zero charge for all 622 of those events, because we only meter what the upstream returns in its usage block — and a request that died mid-generation doesn't return one. So I can show you exactly how often calls fail, and I cannot show you what an abandoned generation cost upstream. That's a real gap in our metering, not a clean bill of health, and it's the number I'd most like to have.\n\nYour own logs probably have the same hole. Fill it before you trust any cost model built on call counts. Count attempts.", "url": "https://wpnews.pro/news/622-of-our-5087-llm-api-calls-never-returned-an-answer", "canonical_source": "https://dev.to/altrouter/622-of-our-5087-llm-api-calls-never-returned-an-answer-ale", "published_at": "2026-09-11 06:01:29+00:00", "updated_at": "2026-09-11 06:25:56.705780+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops", "developer-tools"], "entities": ["gpt-5.6"], "alternates": {"html": "https://wpnews.pro/news/622-of-our-5087-llm-api-calls-never-returned-an-answer", "markdown": "https://wpnews.pro/news/622-of-our-5087-llm-api-calls-never-returned-an-answer.md", "text": "https://wpnews.pro/news/622-of-our-5087-llm-api-calls-never-returned-an-answer.txt", "jsonld": "https://wpnews.pro/news/622-of-our-5087-llm-api-calls-never-returned-an-answer.jsonld"}}