{"slug": "streaming-llm-responses-tokens-arrive-faster-than-you-should-render-them", "title": "Streaming LLM Responses: Tokens Arrive Faster Than You Should Render Them", "summary": "A technical guide published on streaming LLM responses argues that enabling `stream: true` is only the first step, and that production streaming requires three separate pieces: a server relay that wraps the provider's raw stream in a controlled event protocol, a client reader built on `fetch` rather than `EventSource`, and a renderer that batches tokens with `requestAnimationFrame` instead of re-parsing Markdown and rewriting `innerHTML` on every token. The guide's example route handler uses the Anthropic TypeScript SDK with the `claude-sonnet-5` model and `max_tokens` set to 1024, emitting `delta`, `done`, and `error` events, and calls `llm.abort()` in the stream's `cancel()` handler so generation stops when the browser disconnects. It notes that providers can send errors inside the stream — Anthropic's `overloaded_error` is the in-stream equivalent of an HTTP 529 — so failures must arrive as events rather than status codes, and that `done` carries `stop_reason` so the UI can distinguish a finished answer from one cut off at `max_tokens`.", "body_md": "Turning on streaming is usually one flag. Set `stream: true`, and instead of waiting eight seconds for a full answer, the user sees the first words in a few hundred milliseconds. The model isn't any faster. It just feels faster, which is the point.\n\nThen it goes to production. A reverse proxy holds the whole response and sends it at the end. The chat pane stutters, because every token re-parses the entire Markdown document and rewrites `innerHTML`. The \"Stop\" button hides the text but the server keeps generating, so you still pay for the tokens. And when the provider fails halfway through, the HTTP status has already gone out as 200, so your `if (!res.ok)` check never runs. Getting streaming right takes three pieces: the server relay, the client reader, and the renderer.\n\n## The server: relay events, and own the failure path \n\nDon't pipe the provider's raw stream straight to the browser. Wrap it in a small event protocol you control. That keeps your API key on the server, and it gives errors and completion their own events. Here's a route handler using the [Anthropic TypeScript SDK](https://github.com/anthropics/anthropic-sdk-typescript/blob/main/helpers.md). It uses the standard `Request`/` Response` objects, so it works in Next.js route handlers, Nitro/Nuxt, and most edge runtimes:\n\n``` python\nimport Anthropic from \"@anthropic-ai/sdk\";\n\nconst client = new Anthropic();\n\nexport async function POST(req: Request) {\n  const { messages } = await req.json();\n  const encoder = new TextEncoder();\n  const llm = client.messages.stream({\n    model: \"claude-sonnet-5\",\n    max_tokens: 1024,\n    messages,\n  });\n\n  const body = new ReadableStream({\n    start(controller) {\n      const send = (event: string, data: unknown) => {\n        try {\n          controller.enqueue(\n            encoder.encode(\n              `event: ${event}\\ndata: ${JSON.stringify(data)}\\n\\n`,\n            ),\n          );\n        } catch {\n          // client already gone\n        }\n      };\n      llm.on(\"text\", (delta) => send(\"delta\", { text: delta }));\n      llm.on(\"finalMessage\", (msg) =>\n        send(\"done\", { stopReason: msg.stop_reason }),\n      );\n      llm.on(\"error\", () => send(\"error\", { message: \"generation_failed\" }));\n      llm.on(\"end\", () => {\n        try {\n          controller.close();\n        } catch {}\n      });\n    },\n    cancel() {\n      llm.abort(); // client disconnected: stop generating (and paying)\n    },\n  });\n\n  return new Response(body, {\n    headers: {\n      \"Content-Type\": \"text/event-stream\",\n      \"Cache-Control\": \"no-cache, no-transform\",\n      \"X-Accel-Buffering\": \"no\", // tells nginx not to buffer this response\n    },\n  });\n}\n```\n\nThere are three details here that people tend to learn the hard way. First, the provider can send an error *inside* the stream: Anthropic's [streaming docs](https://platform.claude.com/docs/en/build-with-claude/streaming) give `overloaded_error` as the in-stream version of an HTTP 529. So \"error\" has to be an event, not a status code. Second, `cancel()` runs when the browser goes away, and `llm.abort()` closes the upstream request so generation actually stops. Third, `done` carries `stop_reason`, so the UI can tell a finished answer from one cut off at `max_tokens`.\n\n## The client: read with fetch, batch with requestAnimationFrame \n\n`EventSource` only does GET requests and can't send a body, so it doesn't fit a chat endpoint. Read the response with `fetch` and parse the frames yourself. It's about 20 lines. The part that matters is that a token doesn't trigger a render. Tokens pile up in a buffer, and that buffer gets flushed once per frame:\n\n```\nexport async function streamChat(\n  messages: unknown[],\n  onText: (chunk: string) => void,\n  signal: AbortSignal,\n) {\n  const res = await fetch(\"/api/chat\", {\n    method: \"POST\",\n    headers: { \"Content-Type\": \"application/json\" },\n    body: JSON.stringify({ messages }),\n    signal,\n  });\n  if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);\n\n  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();\n  let buffer = \"\";\n  let pending = \"\";\n  let frame = 0;\n  const flush = () => {\n    frame = 0;\n    onText(pending);\n    pending = \"\";\n  };\n\n  while (true) {\n    const { value, done } = await reader.read();\n    if (done) break;\n    buffer += value;\n    const frames = buffer.split(\"\\n\\n\");\n    buffer = frames.pop() ?? \"\"; // keep the incomplete frame for the next read\n    for (const raw of frames) {\n      const event = raw.match(/^event: (.*)$/m)?.[1];\n      const data = JSON.parse(raw.match(/^data: (.*)$/m)?.[1] ?? \"{}\");\n      if (event === \"delta\") {\n        pending += data.text;\n        frame ||= requestAnimationFrame(flush);\n      }\n      if (event === \"error\") throw new Error(data.message);\n    }\n  }\n  if (frame) cancelAnimationFrame(frame);\n  if (pending) flush();\n}\n```\n\nA network chunk doesn't always line up with an event boundary, which is why the last piece of `buffer` is held back until the next read. The `signal` comes from an `AbortController` attached to your Stop button. Aborting the fetch closes the connection, which runs the server's `cancel()`, which aborts the upstream call. One click stops the whole chain.\n\n## Rendering: append, don't re-parse \n\nFor plain text, the [Chrome team's guidance](https://developer.chrome.com/docs/ai/render-llm-responses) is to use `element.append(chunk)` rather than `textContent +=`, because the second one throws away and rebuilds the text node on every update. Markdown is harder. The naive loop, `innerHTML = marked.parse(allTextSoFar)`, re-parses and re-renders the whole document on every flush. On a long answer with syntax-highlighted code blocks, that cost grows with every update.\n\nIt also has a security problem. Model output is untrusted input: a prompt injection can get the model to emit `<img onerror=...>`. The same guidance recommends a streaming Markdown parser such as [streaming-markdown](https://github.com/thetarnav/streaming-markdown), which appends nodes instead of replacing them. Pair it with a sanitizer like DOMPurify, run over the accumulated text, and stop rendering if it removes anything.\n\nTwo more things are easy to skip. Set `aria-busy=\"true\"` on the response region while it streams and remove it on `done`, so screen readers don't try to announce every partial update (support varies, so test with the readers your users have). And when the stream ends on an `error` event or a `max_tokens` stop, keep the partial text visible, label it as incomplete, and offer a retry. Silently replacing it with an error banner throws away an answer the user already started reading.\n\nStreaming is a UX feature, so test it the way users experience it. Throttle the network in DevTools, put your real proxy in front of the dev server, and turn on Paint flashing while a long answer comes in. If the whole pane flashes on every token, the renderer is the bottleneck, not the model.", "url": "https://wpnews.pro/news/streaming-llm-responses-tokens-arrive-faster-than-you-should-render-them", "canonical_source": "https://danholloran.me/posts/streaming-llm-responses-tokens-arrive-faster-than-you-should-render-them", "published_at": "2026-11-21 10:16:51+00:00", "updated_at": "2026-11-21 14:43:38.670546+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "generative-ai"], "entities": ["Anthropic", "Anthropic TypeScript SDK", "claude-sonnet-5", "Next.js", "Nitro", "Nuxt", "EventSource"], "also_reported_by": [], "takeaway": "Batch streamed tokens in a buffer flushed once per animation frame instead of re-rendering per token, and call `llm.abort()` in the ReadableStream's `cancel()` handler so a disconnected client stops generation and token billing.", "recommendation": {"advice": "Buffer streamed tokens and flush once per requestAnimationFrame rather than re-parsing Markdown per token, and abort the upstream request in the stream's cancel() handler so disconnected clients stop paying for tokens.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/streaming-llm-responses-tokens-arrive-faster-than-you-should-render-them", "markdown": "https://wpnews.pro/news/streaming-llm-responses-tokens-arrive-faster-than-you-should-render-them.md", "text": "https://wpnews.pro/news/streaming-llm-responses-tokens-arrive-faster-than-you-should-render-them.txt", "jsonld": "https://wpnews.pro/news/streaming-llm-responses-tokens-arrive-faster-than-you-should-render-them.jsonld"}}