{"slug": "ai-api-streaming-not-working-debug-sse-and-nginx-buffering", "title": "AI API Streaming Not Working? Debug SSE and Nginx Buffering", "summary": "A developer published a diagnostic guide for debugging AI API streaming failures, using a small local Server-Sent Events (SSE) endpoint built with Python's standard library to isolate where incremental bytes stop arriving. The guide recommends comparing a direct localhost request against the same endpoint routed through a staging proxy to distinguish application, reverse-proxy/CDN, and browser buffering, and notes that an HTTP 200 only confirms headers were sent, not that each event reached the client.", "body_md": "You enable streaming in an AI API request. The server returns 200. Then nothing appears for several seconds, and the entire answer lands at once.\n\nBefore changing models or increasing timeouts, find out where the bytes stop arriving incrementally. The model service, your application, a reverse proxy, and the browser can each turn a stream into a single response.\n\nThis guide uses a small local SSE endpoint to separate those layers. It does not require an API key or make a paid model request.\n\nA long wait before the first visible text is not enough to diagnose a buffering problem. The upstream service may still be preparing an answer. Tool calls and other non-text events can also arrive before any displayable text.\n\nA more useful clue is this: the same request arrives progressively when you call the application directly, but arrives in a burst through your public URL. That comparison points toward something between the application and the client.\n\nAn HTTP 200 status only confirms that response headers were sent. It does not prove that the complete generation succeeded, or that the browser received each event as it was produced.\n\nSave this as `sse_demo.py`. It sends five events, roughly 400 milliseconds apart, using only Python’s standard library. It binds to localhost for a local diagnostic test.\n\n``` python\nfrom http.server import BaseHTTPRequestHandler, ThreadingHTTPServer\nimport json\nimport time\n\nclass Handler(BaseHTTPRequestHandler):\n    protocol_version = \"HTTP/1.1\"\n\n    def do_GET(self):\n        if self.path != \"/demo/events\":\n            self.send_error(404)\n            return\n        self.send_response(200)\n        self.send_header(\"Content-Type\", \"text/event-stream; charset=utf-8\")\n        self.send_header(\"Cache-Control\", \"no-cache, no-transform\")\n        self.send_header(\"Connection\", \"close\")\n        self.end_headers()\n        self.close_connection = True\n        try:\n            for i in range(5):\n                data = json.dumps({\"step\": i + 1})\n                self.wfile.write(f\"data: {data}\\n\\n\".encode(\"utf-8\"))\n                self.wfile.flush()\n                time.sleep(0.4)\n        except (BrokenPipeError, ConnectionResetError):\n            pass\n\nif __name__ == \"__main__\":\n    ThreadingHTTPServer((\"127.0.0.1\", 9000), Handler).serve_forever()\n```\n\nRun the server in one terminal:\n\n```\npython3 sse_demo.py\n```\n\nIn another terminal:\n\n```\ncurl --noproxy \"*\" --no-buffer --include --max-time 10 \\\n  http://127.0.0.1:9000/demo/events\n```\n\nYou should see five separate `data:` messages appear over about two seconds. `--no-buffer`, also spelled `-N`, disables curl’s output buffering. It cannot disable buffering inside your application or proxy.\n\nEach SSE event ends with a blank line: `\\n\\n`. A server that writes the payload but omits the event separator may be sending bytes while an SSE parser waits for a complete event. The demo also explicitly flushes each write.\n\nThis checks the downstream transport only. A real AI endpoint still needs its own test, with the documented HTTP method, authentication, request body, model, and streaming option.\n\nIf you control a staging proxy, route this endpoint through the same layers used by your streaming API. Compare the direct localhost request with:\n\n```\ncurl --no-buffer --include --max-time 10 \\\n  https://YOUR_STAGING_HOST/demo/events\n```\n\nUse your own staging hostname; this path is a diagnostic example, not an endpoint offered by every API service.\n\n| What you observe | Where to investigate next | \n|---|---|\n| The local demo arrives in a burst | The handler, flushing, or the local client | \n| Direct access streams; public access bursts | Reverse proxy, CDN, middleware, or compression | \n| Public curl streams; the web UI bursts | Browser response handling and UI updates | \n| The demo streams everywhere; a model request starts late | The real request’s upstream timing and event types | \n\nA browser Network panel entry staying “pending” is normal while a stream is open. Look at incoming events and timing rather than treating “pending” as a failure.\n\nNginx enables proxy response buffering by default. A minimal diagnostic location, inside an existing `server` block, looks like this:\n\n```\nlocation = /demo/events {\n    proxy_pass http://127.0.0.1:9000;\n    proxy_http_version 1.1;\n    proxy_set_header Connection \"\";\n\n    proxy_buffering off;\n    proxy_cache off;\n    proxy_read_timeout 120s;\n\n}\n```\n\nApply the relevant settings to the actual streaming route after verifying the diagnostic endpoint. The empty `Connection` header here is for the Nginx-to-upstream hop; this example uses SSE over HTTP, not a WebSocket upgrade.\n\nValidate the configuration with `nginx -t` before reloading it. Test in staging and preserve the authentication, limits, and other controls already on your production route.\n\nTwo similarly named directives are easy to mix up: `proxy_buffering` controls the upstream response, while `proxy_request_buffering` controls the client’s request body. Disabling request buffering alone will not fix an answer that arrives all at once.\n\nAn application can alternatively send `X-Accel-Buffering: no` in its upstream response. Nginx must actually process that header; `proxy_ignore_headers` can disable its effect. Merely adding the header to the outgoing response with `add_header` is not equivalent to disabling Nginx’s proxy buffering.\n\nIf a CDN, another proxy, or compression middleware sits in the path, test that layer too. Turning off buffering in one hop does not change every other hop.\n\nThese calls wait for the response body to finish:\n\n``` js\nconst text = await response.text();\n\nconst data = await response.json();\n```\n\nFor a streaming interface, read `response.body` progressively and decode UTF-8 incrementally. Use an SSE parser that retains incomplete events between reads. A network chunk is not necessarily one token, one JSON object, or one complete SSE event.\n\nNative `EventSource` works well for a GET-based SSE endpoint such as this demo. It does not expose a general POST body or arbitrary Authorization headers. For a model API that requires POST, a fetch-based streaming client is often appropriate. Keep provider credentials in your server-side integration rather than embedding them in a public frontend.\n\nAlso inspect your rendering logic. If code accumulates every delta and updates the UI only after the read loop ends, the network can be streaming correctly while the page still looks buffered.\n\nFor Nginx, `proxy_read_timeout` measures the gap between successive reads from the upstream, not the total duration of a generation. A 120-second value therefore does not impose a universal two-minute generation limit.\n\nFor idle connections you control, an SSE comment such as `: keepalive\\n\\n` can act as a heartbeat. Check the actual idle limits of each proxy and hosting platform. Heartbeats do not mean that the model has produced a token, and they do not bypass a platform’s absolute execution limit.\n\nHandle disconnects, incomplete responses, and cancellation explicitly. Replaying a whole request after displaying partial text can duplicate output and may incur another charge. Do not treat every stream failure as a reason to retry automatically.\n\nWhen recording timings, distinguish the first response byte, the first complete event, the first text delta, and completion. Those timestamps answer different questions.\n\nKeep the report small: the request time, a non-secret request ID if available, HTTP status, response Content-Type, which route was tested, and whether events arrived gradually or in a burst. Remove API keys, Authorization headers, and sensitive prompts.\n\nThe most useful debugging result is usually a comparison: “localhost streams, public URL buffers” or “curl streams, the UI waits until completion.” That narrows the next change much more than “the AI is slow.”\n\nWritten for the [YW Code API](https://ywcode.top/) developer notes. YW Code is an independent AI API gateway. The local example above runs without an account and makes no model API calls.", "url": "https://wpnews.pro/news/ai-api-streaming-not-working-debug-sse-and-nginx-buffering", "canonical_source": "https://dev.to/_fde76ff2d01974642930b/ai-api-streaming-not-working-debug-sse-and-nginx-buffering-3ina", "published_at": "2026-11-09 03:49:34+00:00", "updated_at": "2026-11-09 03:52:16.912572+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["curl", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-api-streaming-not-working-debug-sse-and-nginx-buffering", "markdown": "https://wpnews.pro/news/ai-api-streaming-not-working-debug-sse-and-nginx-buffering.md", "text": "https://wpnews.pro/news/ai-api-streaming-not-working-debug-sse-and-nginx-buffering.txt", "jsonld": "https://wpnews.pro/news/ai-api-streaming-not-working-debug-sse-and-nginx-buffering.jsonld"}}