You enable streaming in an AI API request. The server returns 200. Then nothing appears for several seconds, and the entire answer lands at once.
Before changing models or increasing timeouts, find out where the bytes stop arriving incrementally. The model service, your application, a reverse proxy, and the browser can each turn a stream into a single response.
This guide uses a small local SSE endpoint to separate those layers. It does not require an API key or make a paid model request.
A long wait before the first visible text is not enough to diagnose a buffering problem. The upstream service may still be preparing an answer. Tool calls and other non-text events can also arrive before any displayable text.
A more useful clue is this: the same request arrives progressively when you call the application directly, but arrives in a burst through your public URL. That comparison points toward something between the application and the client.
An HTTP 200 status only confirms that response headers were sent. It does not prove that the complete generation succeeded, or that the browser received each event as it was produced.
Save this as sse_demo.py. It sends five events, roughly 400 milliseconds apart, using only Python’s standard library. It binds to localhost for a local diagnostic test.
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import time
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def do_GET(self):
if self.path != "/demo/events":
self.send_error(404)
return
self.send_response(200)
self.send_header("Content-Type", "text/event-stream; charset=utf-8")
self.send_header("Cache-Control", "no-cache, no-transform")
self.send_header("Connection", "close")
self.end_headers()
self.close_connection = True
try:
for i in range(5):
data = json.dumps({"step": i + 1})
self.wfile.write(f"data: {data}\n\n".encode("utf-8"))
self.wfile.flush()
time.sleep(0.4)
except (BrokenPipeError, ConnectionResetError):
pass
if __name__ == "__main__":
ThreadingHTTPServer(("127.0.0.1", 9000), Handler).serve_forever()
Run the server in one terminal:
python3 sse_demo.py
In another terminal:
curl --noproxy "*" --no-buffer --include --max-time 10 \
http://127.0.0.1:9000/demo/events
You should see five separate data: messages appear over about two seconds. --no-buffer, also spelled -N, disables curl’s output buffering. It cannot disable buffering inside your application or proxy.
Each SSE event ends with a blank line: \n\n. A server that writes the payload but omits the event separator may be sending bytes while an SSE parser waits for a complete event. The demo also explicitly flushes each write.
This checks the downstream transport only. A real AI endpoint still needs its own test, with the documented HTTP method, authentication, request body, model, and streaming option.
If you control a staging proxy, route this endpoint through the same layers used by your streaming API. Compare the direct localhost request with:
curl --no-buffer --include --max-time 10 \
https://YOUR_STAGING_HOST/demo/events
Use your own staging hostname; this path is a diagnostic example, not an endpoint offered by every API service.
| What you observe | Where to investigate next |
|---|---|
| The local demo arrives in a burst | The handler, flushing, or the local client |
| Direct access streams; public access bursts | Reverse proxy, CDN, middleware, or compression |
| Public curl streams; the web UI bursts | Browser response handling and UI updates |
| The demo streams everywhere; a model request starts late | The real request’s upstream timing and event types |
A browser Network panel entry staying “pending” is normal while a stream is open. Look at incoming events and timing rather than treating “pending” as a failure.
Nginx enables proxy response buffering by default. A minimal diagnostic location, inside an existing server block, looks like this:
location = /demo/events {
proxy_pass http://127.0.0.1:9000;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 120s;
}
Apply the relevant settings to the actual streaming route after verifying the diagnostic endpoint. The empty Connection header here is for the Nginx-to-upstream hop; this example uses SSE over HTTP, not a WebSocket upgrade.
Validate the configuration with nginx -t before re it. Test in staging and preserve the authentication, limits, and other controls already on your production route.
Two similarly named directives are easy to mix up: proxy_buffering controls the upstream response, while proxy_request_buffering controls the client’s request body. Disabling request buffering alone will not fix an answer that arrives all at once.
An application can alternatively send X-Accel-Buffering: no in its upstream response. Nginx must actually process that header; proxy_ignore_headers can disable its effect. Merely adding the header to the outgoing response with add_header is not equivalent to disabling Nginx’s proxy buffering.
If a CDN, another proxy, or compression middleware sits in the path, test that layer too. Turning off buffering in one hop does not change every other hop.
These calls wait for the response body to finish:
const text = await response.text();
const data = await response.json();
For a streaming interface, read response.body progressively and decode UTF-8 incrementally. Use an SSE parser that retains incomplete events between reads. A network chunk is not necessarily one token, one JSON object, or one complete SSE event.
Native EventSource works well for a GET-based SSE endpoint such as this demo. It does not expose a general POST body or arbitrary Authorization headers. For a model API that requires POST, a fetch-based streaming client is often appropriate. Keep provider credentials in your server-side integration rather than embedding them in a public frontend.
Also inspect your rendering logic. If code accumulates every delta and updates the UI only after the read loop ends, the network can be streaming correctly while the page still looks buffered.
For Nginx, proxy_read_timeout measures the gap between successive reads from the upstream, not the total duration of a generation. A 120-second value therefore does not impose a universal two-minute generation limit.
For idle connections you control, an SSE comment such as : keepalive\n\n can act as a heartbeat. Check the actual idle limits of each proxy and hosting platform. Heartbeats do not mean that the model has produced a token, and they do not bypass a platform’s absolute execution limit.
Handle disconnects, incomplete responses, and cancellation explicitly. Replaying a whole request after displaying partial text can duplicate output and may incur another charge. Do not treat every stream failure as a reason to retry automatically.
When recording timings, distinguish the first response byte, the first complete event, the first text delta, and completion. Those timestamps answer different questions.
Keep the report small: the request time, a non-secret request ID if available, HTTP status, response Content-Type, which route was tested, and whether events arrived gradually or in a burst. Remove API keys, Authorization headers, and sensitive prompts.
The most useful debugging result is usually a comparison: “localhost streams, public URL buffers” or “curl streams, the UI waits until completion.” That narrows the next change much more than “the AI is slow.”
Written for the YW Code API developer notes. YW Code is an independent AI API gateway. The local example above runs without an account and makes no model API calls.