Streaming LLM Responses: Tokens Arrive Faster Than You Should Render Them A technical guide published on streaming LLM responses argues that enabling `stream: true` is only the first step, and that production streaming requires three separate pieces: a server relay that wraps the provider's raw stream in a controlled event protocol, a client reader built on `fetch` rather than `EventSource`, and a renderer that batches tokens with `requestAnimationFrame` instead of re-parsing Markdown and rewriting `innerHTML` on every token. The guide's example route handler uses the Anthropic TypeScript SDK with the `claude-sonnet-5` model and `max_tokens` set to 1024, emitting `delta`, `done`, and `error` events, and calls `llm.abort()` in the stream's `cancel()` handler so generation stops when the browser disconnects. It notes that providers can send errors inside the stream — Anthropic's `overloaded_error` is the in-stream equivalent of an HTTP 529 — so failures must arrive as events rather than status codes, and that `done` carries `stop_reason` so the UI can distinguish a finished answer from one cut off at `max_tokens`. Turning on streaming is usually one flag. Set stream: true , and instead of waiting eight seconds for a full answer, the user sees the first words in a few hundred milliseconds. The model isn't any faster. It just feels faster, which is the point. Then it goes to production. A reverse proxy holds the whole response and sends it at the end. The chat pane stutters, because every token re-parses the entire Markdown document and rewrites innerHTML . The "Stop" button hides the text but the server keeps generating, so you still pay for the tokens. And when the provider fails halfway through, the HTTP status has already gone out as 200, so your if res.ok check never runs. Getting streaming right takes three pieces: the server relay, the client reader, and the renderer. The server: relay events, and own the failure path Don't pipe the provider's raw stream straight to the browser. Wrap it in a small event protocol you control. That keeps your API key on the server, and it gives errors and completion their own events. Here's a route handler using the Anthropic TypeScript SDK https://github.com/anthropics/anthropic-sdk-typescript/blob/main/helpers.md . It uses the standard Request / Response objects, so it works in Next.js route handlers, Nitro/Nuxt, and most edge runtimes: python import Anthropic from "@anthropic-ai/sdk"; const client = new Anthropic ; export async function POST req: Request { const { messages } = await req.json ; const encoder = new TextEncoder ; const llm = client.messages.stream { model: "claude-sonnet-5", max tokens: 1024, messages, } ; const body = new ReadableStream { start controller { const send = event: string, data: unknown = { try { controller.enqueue encoder.encode event: ${event}\ndata: ${JSON.stringify data }\n\n , , ; } catch { // client already gone } }; llm.on "text", delta = send "delta", { text: delta } ; llm.on "finalMessage", msg = send "done", { stopReason: msg.stop reason } , ; llm.on "error", = send "error", { message: "generation failed" } ; llm.on "end", = { try { controller.close ; } catch {} } ; }, cancel { llm.abort ; // client disconnected: stop generating and paying }, } ; return new Response body, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache, no-transform", "X-Accel-Buffering": "no", // tells nginx not to buffer this response }, } ; } There are three details here that people tend to learn the hard way. First, the provider can send an error inside the stream: Anthropic's streaming docs https://platform.claude.com/docs/en/build-with-claude/streaming give overloaded error as the in-stream version of an HTTP 529. So "error" has to be an event, not a status code. Second, cancel runs when the browser goes away, and llm.abort closes the upstream request so generation actually stops. Third, done carries stop reason , so the UI can tell a finished answer from one cut off at max tokens . The client: read with fetch, batch with requestAnimationFrame EventSource only does GET requests and can't send a body, so it doesn't fit a chat endpoint. Read the response with fetch and parse the frames yourself. It's about 20 lines. The part that matters is that a token doesn't trigger a render. Tokens pile up in a buffer, and that buffer gets flushed once per frame: export async function streamChat messages: unknown , onText: chunk: string = void, signal: AbortSignal, { const res = await fetch "/api/chat", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify { messages } , signal, } ; if res.ok || res.body throw new Error HTTP ${res.status} ; const reader = res.body.pipeThrough new TextDecoderStream .getReader ; let buffer = ""; let pending = ""; let frame = 0; const flush = = { frame = 0; onText pending ; pending = ""; }; while true { const { value, done } = await reader.read ; if done break; buffer += value; const frames = buffer.split "\n\n" ; buffer = frames.pop ?? ""; // keep the incomplete frame for the next read for const raw of frames { const event = raw.match /^event: . $/m ?. 1 ; const data = JSON.parse raw.match /^data: . $/m ?. 1 ?? "{}" ; if event === "delta" { pending += data.text; frame ||= requestAnimationFrame flush ; } if event === "error" throw new Error data.message ; } } if frame cancelAnimationFrame frame ; if pending flush ; } A network chunk doesn't always line up with an event boundary, which is why the last piece of buffer is held back until the next read. The signal comes from an AbortController attached to your Stop button. Aborting the fetch closes the connection, which runs the server's cancel , which aborts the upstream call. One click stops the whole chain. Rendering: append, don't re-parse For plain text, the Chrome team's guidance https://developer.chrome.com/docs/ai/render-llm-responses is to use element.append chunk rather than textContent += , because the second one throws away and rebuilds the text node on every update. Markdown is harder. The naive loop, innerHTML = marked.parse allTextSoFar , re-parses and re-renders the whole document on every flush. On a long answer with syntax-highlighted code blocks, that cost grows with every update. It also has a security problem. Model output is untrusted input: a prompt injection can get the model to emit