Streaming LLM output through a Next.js proxy: why you only get stream_end

Ali Ahmed documented the specific configuration needed to stream LLM Server-Sent Events through a Next.js route handler without buffering, drawn from building Turbo Alan Refiner, a multi-pass LLM text-refinement app. The setup requires the Node.js runtime, force-dynamic rendering, no-cache/no-transform and X-Accel-Buffering: no headers, byte-by-byte forwarding via a ReadableStream reader, and a client that buffers incomplete lines and treats only terminal events as the end. Ahmed notes that gzip-compressing proxies reintroduce buffering and recommends DevTools' EventStream view to distinguish client-side from upstream buffering.

When you proxy an LLM backend's Server-Sent Events SSE through a Next.js route handler, the usual failure is that nothing arrives until the end, and then everything arrives at once, or the client only ever sees the final stream end . Something between the backend and the browser is buffering . The fix is a handful of specific things: the Node.js runtime, dynamic = 'force-dynamic' , the right headers Cache-Control: no-cache, no-transform and X-Accel-Buffering: no , passing bytes through as they arrive, and a client that parses SSE frames properly. I learned these building Turbo Alan Refiner , a multi-pass LLM text-refinement app frontend source https://github.com/QuantumBreakz/TurboAlanRefinerFrontend . Routing every backend call through /api/ route handlers avoids CORS, keeps tokens server-side, and lets you normalise the request. The cost is that your route is now a hop in a streaming pipeline, and every hop is a chance to buffer. ts title="Route segment config" export const runtime = "nodejs"; export const dynamic = "force-dynamic"; export const maxDuration = 300; runtime = "nodejs" gives full streaming support without edge-runtime limits. dynamic = "force-dynamic" stops Next.js caching or pre-rendering the route. maxDuration = 300 lets long refinements outlive the default function timeout. Then return a ReadableStream that forwards the upstream body as it arrives, with these headers: ts title="Streaming response headers" return new Response stream, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache, no-transform", Connection: "keep-alive", "X-Accel-Buffering": "no", }, } ; X-Accel-Buffering: no tells nginx-style proxies not to buffer. no-transform matters more than it looks. A proxy that gzips the response has to collect bytes before it can compress them, which is exactly the buffering you're trying to avoid. getReader and decode with a streaming TextDecoder . await upstream.text , which waits for the whole response. data: {...} and your proxy prefixes data: again, the client receives data: data: {...} . data: {"type":"error",...} frame so the UI can show it, instead of the stream just stopping. AbortController so closing the tab cancels the upstream request, rather than leaving an LLM job running for nobody. Usually one of three things: Something buffered the stream missing headers, the wrong runtime, a proxy in front , so all the events arrived together and the UI only rendered the last one. The client parsed chunks, not lines. A network chunk can end in the middle of an event. Keep the incomplete tail in a buffer: ts title="Client: keep the incomplete last line" buffer += decoder.decode chunk, { stream: true } ; const lines = buffer.split "\n" ; buffer = lines.pop ?? ""; 3. The client stopped too early. Treat only true terminal events complete , error as the end. Keep processing everything already buffered before you close. Be lenient about input, too: strip a doubled data: data: prefix, and accept raw JSON lines as well as SSE-formatted ones. How do you debug it? Open DevTools → Network → the streaming request → EventStream . If events show up one by one there but the UI updates all at once, the bug is in your client. If they all show up together, something upstream is buffering: check the headers, the runtime, and any proxy or CDN in front. Related: what a serverless cold start really is https://aliahmed.si/blog/serverless-cold-starts-explained/ long-lived streams and function timeouts run into the same platform limits . --- Written by Ali Ahmed · Original article https://aliahmed.si/blog/nextjs-sse-proxy-streaming/