{"slug": "resumable-token-streaming-why-ai-streams-break-on-reconnect", "title": "Resumable token streaming: why AI streams break on reconnect", "summary": "Resumable token streaming requires separating generation, storage, and delivery so a dropped connection only interrupts delivery rather than losing the AI response, according to an Ably blog post on why AI streams break on reconnect. The post states that resuming a token stream needs four things: generation that outlives the connection, retained output, a stable session ID, and ordered catch-up, and notes that AWS Application Load Balancers close idle connections after 60 seconds by default. It recommends storing each response as one growing message so a returning client gets the answer so far in a single request instead of replaying every token.", "body_md": "Refresh the page, lose signal, switch tabs. Users expect the AI conversation to still be there when they return, with the answer picking up where it left off. Resumable token streaming makes that possible: a returning client catches up with the in-progress response instead of starting over.\n\nMaking a stream resumable is an architecture decision. Starting generation, storing the response, and delivering it need to happen in separate parts of your stack. A dropped connection then only interrupts delivery, and the response waits where the client can get it back.\n\nThis post covers why AI streams break on reconnect, what a resumable architecture needs, and the options for building one.\n\n## Key takeaways\n\n- An AI response is lost on reload or disconnect when it exists only in the connection and the browser.\n- A resumable architecture separates starting generation, storing the response, and delivering it, so a dropped connection only interrupts delivery.\n- Resuming a token stream takes four things: generation that outlives the connection, retained output, a stable session ID, and ordered catch-up.\n- Store each response as one growing message, so a returning client gets the answer so far in one request instead of replaying every token.\n- Each failure has its own owner: the transport resumes brief drops, your app restores from history after a reload, and a workflow engine retries crashes.\n- A framework helper can cover short answers on one device; a managed transport fits when responses run long or span tabs and devices.\n\n## Why do AI token streams break?\n\nAI token streams break when a connection closes, and the application cannot reconnect the client to the response already in progress. Continuity can fail in three places: at the connection, in state held only by the browser, and in a backend that cannot restore missed output.\n\n### A closed connection interrupts live delivery\n\nIf you're streaming an AI response over a standard HTTP connection and it drops, that request ends. HTTP does not resume a half-finished response on its own. A [__WebSocket connection__](https://ably.com/topic/websockets) also closes when it drops; either approach needs recovery logic to restore what the client missed.\n\nIn enterprise environments, the problem is worse. Proxies and load balancers close connections that look idle. An [__AWS Application Load Balancer__](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-load-balancer-attributes.html#connection-idle-timeout), for example, closes them after 60 seconds by default unless the timeout is changed.\n\nFor timeout values by platform and the server-side ping fix, see [__WebSocket reconnection in AI agents__](https://ably.com/blog/websocket-reconnection-timeouts-ai-agents).\n\n### Browser-only state disappears on reload\n\nIf the browser tab crashes or reloads, any part of the response held only in browser memory is lost. The application can bring it back only if a copy of the in-progress response was kept outside the browser. Otherwise, the user faces a hard reset.\n\nA network blip does similar damage when no copy was kept: the browser keeps what it received, but the rest of the response is lost. The user may need to send the prompt again and wait, and you pay for another model call while the output already generated goes to waste.\n\n### A backend without retained state cannot restore missed output\n\nMany backends treat streaming as fire-and-forget: tokens go out and aren't stored. When a client reconnects and asks what it missed, the server has no record to answer from.\n\n**Restoring missed output needs three things that HTTP streaming leaves to your application:**\n\n1. A way to identify the response the client was watching\n2. Output that is retained somewhere other than the connection\n3. An ordered way to send back only what the client is missing\n\n[__SSE's Last-Event-ID header__](https://ably.com/topic/server-sent-events) tells the server where a client stopped, but the server still has to supply all three.\n\nResponse identity, retained output, and ordered catch-up are harder to add after launch than to design in. All three touch storage, every client's reconnect logic, and how history is read back. The storage choice is the hardest to reverse: moving a live system from one record per token to one record per response is [__a migration project of its own__](https://ably.com/blog/resume-tokens-last-event-id-llm-streaming-reconnection). Switching from SSE to WebSockets doesn't remove the need for any of them, because [__a WebSocket reconnect restores the connection, not the output generated while it was down__](https://ably.com/blog/is-websockets-enough-for-ai-chat).\n\n## What does a resumable streaming architecture look like?\n\nA closed connection, browser-only state, and a backend with nothing retained share one root cause. A basic HTTP stream ties three jobs to one connection (starting generation, keeping the only copy of the output, and delivering it to the user), so when the connection closes, all three can fail at once. A resumable architecture gives each job its own part of the stack, and losing the connection then only interrupts delivery.\n\n### The parts of a resumable streaming stack\n\nStarting generation takes two parts, a short HTTP request and the agent, and a workflow engine sits alongside the agent to recover its work after a crash.\n\n### How a response flows through a resumable architecture\n\nWith generation, storage, and delivery separated, every response moves through the same three steps:\n\n1. **The client sends the prompt with a session ID.** The HTTP request returns as soon as the work has started.\n2. **The agent runs the model call** and appends output to the response's message in the shared store.\n3. **Clients subscribe to the session, not the request,** so every tab and device receives the same updates live.\n\n### Which part of your stack recovers the response?\n\nThe flow assumes nothing goes wrong. When something interrupts a response, a [__different part of your stack__](https://ably.com/blog/ai-agent-model-is-fine-but-the-session-is-broken) handles it, and only a crash stops generation unexpectedly:\n\nBecause each interruption needs its own recovery path, treating them as a single \"reconnection\" problem breaks in predictable ways. Reconnect logic on its own survives a network blip but loses the answer on reload, and re-running the model on every failure means paying for a second model call each time.\n\n## What does it take to resume a token stream after a disconnect?\n\nSplitting the stack only helps if each part is built to survive a disconnect. That takes generation that keeps running when the client disconnects, plus the three things a backend usually lacks: retained output, a stable session ID, and ordered catch-up. HTTP streaming and raw WebSocket connections [__leave all four to your application__](https://ably.com/docs/ai-transport/why/http-streaming-and-ai).\n\n### 1. Generation that keeps running when the client disconnects\n\n**The problem:** If generation runs inside the HTTP request that streams the response, a dropped connection can stop generation too. The user then has to send the prompt again, and you pay for a second model call.\n\n**What you need:** A model call that keeps running when the client disconnects. This is your application's responsibility, because a transport layer can deliver output but can't keep your code running.\n\n**How to build it:** Start the model call outside the streaming request, for example in a background job, and publish its output to a store the client can read from. On serverless platforms, the function must also stay alive after the response is sent: in Next.js, [__after()__](https://nextjs.org/docs/app/api-reference/functions/after) keeps the function running up to [__the route's maximum duration__](https://nextjs.org/docs/app/api-reference/file-conventions/route-segment-config/maxDuration).\n\n**In production:** Closing the connection no longer stops generation, and that independence is what makes recovery possible. But closing the tab, or a Stop button that only closes the stream, now leaves the model running and still using tokens. To let users stop a response, [__send a separate stop signal__](https://ably.com/blog/stop-vs-disconnect-canceling-ai-streaming) that reaches the running model call.\n\n### 2. Retained output that a reconnecting client can replay\n\n**The problem:** If output only travels down the connection, anything generated while a client is disconnected is lost. A reconnecting client has nothing to catch up from, even if generation kept running.\n\n**What you need:** A stored copy of the response that outlives the connection and supports a client returning in one of two ways:\n\n- **After a brief drop,** the client picks up from the last update it received and gets everything published during the gap, in order.\n- **After a longer absence or a reload,** the client reads the response from history instead, because the connection can no longer replay what it missed.\n\n**How to build it:** Store the message, not the tokens: keep each response as a single message that grows as the model generates it. Whichever way a client returns, it then gets the response so far in one request and the rest live. The alternative approaches each cost more, as [__A token is not a message__](https://ably.com/blog/store-streamed-ai-responses) shows: a log of separate chunks makes [__every client rebuild the response__](https://ably.com/blog/store-streamed-ai-responses), and rewriting a database row for every token adds a write per token.\n\n**In production:** Retention is a trade-off. Keep output too briefly and returning users find nothing, but keep it longer and you pay to store it.\n\n### 3. A stable session ID that survives reconnects\n\n**The problem:** Retained output only helps if a returning client can find it. After a reload or on a second device, the original connection is gone, and with it the only link to the response the client was watching.\n\n**What you need:** An ID for each conversation that doesn't depend on any one connection.\n\n**How to build it:** Create the ID when the conversation starts and store it with the retained output. Keep a copy on the client, for example in the URL, so a reloaded page, a second tab, or another device can present the same ID and rejoin the in-progress answer.\n\n**In production:**\n\n- **The ID must resolve on any server.** With more than one backend instance, a reconnect can land on a server that has never seen the session, so session state has to live in[__shared storage, not one instance's memory__](https://ably.com/blog/resume-tokens-last-event-id-llm-streaming-reconnection) .\n- **Holding the ID mustn't be enough to join.** If any request carrying a valid ID can reattach, a leaked ID exposes the conversation, so[__check that the user is allowed into that session__](https://ably.com/blog/ai-chat-stream-resumption) before resuming.\n\n### 4. Ordered catch-up without duplicate or missing tokens\n\nThe problem: A connection can drop after the client receives an update but before the server knows it arrived, so the server sends that update again. Other updates can arrive out of order or go missing. The user then sees repeated text or a silent gap, and nobody can tell whether the response is complete.\n\n**What you need:** Catch-up that puts updates in order and skips ones the client already holds.\n\n**How to build it:** Give each update a sequence number. On reconnect, the client sends the last number it processed. The server replays every later update in order, and the client discards any number it has already applied.\n\n**In production:** Storing the response as one growing message makes ordered catch-up simpler. A returning client replaces what it shows with the response so far, then applies only the updates that follow. For how resume tokens and SSE's Last-Event-ID work, and what each costs to build, [__see resume tokens and Last-Event-IDs for LLM streaming__](https://ably.com/blog/resume-tokens-last-event-id-llm-streaming-reconnection).\n\n### Checklist: can your stack resume a token stream after a disconnect?\n\nEach of the four requirements lives in a specific part of a resumable stack. Use the table to review an existing setup, or a design before it ships: a \"no\" in any row shows where a user could lose a response, watch it start over, or see repeated or missing text. The workflow engine sits outside the four requirements: it doesn't help a client reconnect, but it's what recovers the response when the agent itself crashes.\n\n## How can you implement resumable token streaming?\n\nThere are four common ways to meet these requirements. They differ mainly in how much of the stack you build and run yourself.\n\nBuilding it yourself leaves all the work with your team; a framework helper still leaves you the store, stop path, and auth; and a chunk log makes [__every client rebuild the response__](https://ably.com/blog/store-streamed-ai-responses). For short answers to one client on a stable connection, a framework helper may be enough; [__AI chat stream resumption__](https://ably.com/blog/ai-chat-stream-resumption) covers when a Redis buffer will do.\n\nOnce responses run long or users switch tabs and devices, a managed transport such as Ably fits better. It takes the parts with the most edge cases off your team (catch-up while a response is in progress, and delivery to every client), while you keep generation, storage, access, and crash retries.\n\n### Where Ably fits, and what your application still owns\n\nAbly provides the live delivery layer in a resumable stack, while your database stays the record of the conversation. Your agent publishes each response to an [__Ably channel__](https://ably.com/docs/channels) instead of down an HTTP connection. A client that drops, or joins from another device, picks up the same response without the model running again.\n\nAbly delivers each response as one growing message, following a principle Ably CEO and co-founder Matthew O'Riordan describes in [__an interview with The New Stack__](https://thenewstack.io/ably-durable-sessions-ai): a client that reconnects mid-stream should request the current state of the response, not replay every token. A client that joins mid-response gets the answer so far in one piece, then the rest live, instead of waiting while hundreds of tokens replay and your code stitches them back together.\n\nHere's what Ably's realtime infrastructure handles for each of the four requirements and for an agent crash, and what your application still owns:\n\nChoosing Ably means taking on a new dependency: your backend needs an [__Ably auth endpoint__](https://ably.com/docs/ai-transport/setup/authentication), and the conversation's channel needs a [__configuration rule__](https://ably.com/docs/ai-transport/setup/channel-rules). If your answers are short and only ever reach one device, that may not be worth it.\n\nFor longer answers, or users on more than one device, you integrate live catch-up and multi-device delivery rather than build and maintain them. A user who drops mid-response gets their answer back instead of a restart.\n\nTo build a resumable AI chat with Ably, start with the [__quickstart for the Vercel AI SDK__](https://ably.com/docs/ai-transport/streaming/quickstart-vercel), which walks through the auth endpoint, channel rule, agent route, and chat client. The finished app streams each response to every open tab, and a reload reads the conversation back from your own store. To review your own architecture, [__talk to Ably's engineering team__](https://ably.com/contact).\n\n## Frequently asked questions\n\n### What is resumable token streaming?\n\nResumable token streaming lets a client that disconnects mid-response reconnect and receive the rest of the same AI response, without the model running again. It depends on keeping the response outside the connection, plus a way for the client to find it and catch up in order.\n\n### Should generation keep running after the user closes the tab?\n\nUsually, yes, at least for a while: the user may be reloading or switching device, and that shouldn't cost them their answer. But a model call that nobody returns to still uses tokens, so set a limit.\n\nWith Ably, each open tab or device can join the [__conversation's presence list__](https://ably.com/docs/ai-transport/channel/agent-presence), which shows who is connected. The agent can check that list and stop the model call when no user is left.\n\n### When a response stops partway, how do we tell what failed?\n\nCompare what the client shows with what your store holds. If the store has more than the client, delivery failed and catch-up should recover it. If the store's copy also stopped, then generation failed, so check whether the agent is still alive before blaming the model provider.\n\nWith Ably, the agent can join the [__conversation's presence list__](https://ably.com/docs/ai-transport/channel/agent-presence), and Ably removes it about 15 seconds after a crash. Its partial response stays visible to clients during the active turn, still marked as streaming.", "url": "https://wpnews.pro/news/resumable-token-streaming-why-ai-streams-break-on-reconnect", "canonical_source": "https://ably.com/blog/token-streaming-for-ai-ux", "published_at": "2026-10-16 13:57:00+00:00", "updated_at": "2026-10-16 14:49:39.978629+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-agents"], "entities": ["Ably", "AWS Application Load Balancer", "WebSocket", "Server-Sent Events", "Last-Event-ID"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/resumable-token-streaming-why-ai-streams-break-on-reconnect", "markdown": "https://wpnews.pro/news/resumable-token-streaming-why-ai-streams-break-on-reconnect.md", "text": "https://wpnews.pro/news/resumable-token-streaming-why-ai-streams-break-on-reconnect.txt", "jsonld": "https://wpnews.pro/news/resumable-token-streaming-why-ai-streams-break-on-reconnect.jsonld"}}