Backpressure in AI streaming can build anywhere from inference to the client, but delivery is where handling it can change the response itself. The three usual approaches there (buffering, batching, and rewriting a stored record) can change how the response is stored, or what storing it costs. Anyone who joins mid-answer then gets fragments to stitch together, or your database pays for a full write at every flush.
Instead, your backpressure strategy should change only how often tokens move, never how many messages a response becomes. This post tests the three approaches on whether each response stays one message, and introduces an alternative: message appends. For storage patterns in general, see A token is not a message; this post is about what happens under load.
Key takeaways #
- Under load, change only how often tokens move, never how many messages a response becomes.
- Backpressure builds at four points: inference, worker queues, delivery, and clients. Delivery is where the usual fixes also change how the response is stored.
- Buffering, batching, and rewrites each have a cost under load: queued tokens add latency, stored chunks or batches split the response into records, and every flush becomes a full write.
- Test a backpressure strategy in two ways. Can a viewer who can't keep up with the model receive updates at their own pace, without affecting anyone else? And can someone joining mid-answer see the whole response so far?
- Ably message appends keep each response as one growing message, however delivery changes. Your application still controls how much work it accepts and when to cancel it.
How do you handle backpressure in AI streaming at the delivery layer? #
Backpressure is how a system copes when a producer outpaces a consumer: the consumer pushes back, so something waits, slows down, or is dropped. In AI streaming, the producer is the model, and the consumers are every tab, device, and agent watching the response.
Pressure can build at four points between a user's prompt and their screen, and each is handled differently:
Inference and worker queues are capacity decisions your application makes, and client rendering is usually a UI choice. At the delivery layer, teams have mostly relied on three approaches: buffering chunks, batching them, or rewriting one stored record. Handling backpressure well at this layer means changing how often tokens reach each viewer, without changing how many messages a response becomes.
Each works up to a point. The difference shows when the model outpaces a viewer: what each one does to the response, and what it costs.
Two tests for choosing how to deliver AI responses under load #
1. The slow-consumer test
What it checks: live delivery. Can a viewer who can't keep up with the model receive updates at their own pace, without affecting anyone else?
Why it matters: Mobile networks, background tabs, and dashboards that follow many conversations all receive updates more slowly than the model produces them. If one viewer's backlog grows without limit, memory and latency grow with it. Changing what's stored to cope makes every viewer pay for one slow connection.
How to run it: Stream a response to several clients and throttle one, for example with network throttling in browser developer tools. Watch that client's lag, your server's queue depth, and what the other clients and your store end up with. The slow viewer's lag should stay bounded, and everything else should match a run without it.
2. The late-joiner test
What it checks: catch-up. Does someone who joins mid-answer see the whole response so far?
Why it matters: Users reload pages and switch devices, and supervisors or human agents join live conversations. If a late joiner sees a blank screen or a broken response, the user waits for the model to finish, or asks again and pays for a second model call.
How to run it: Start a response that streams for 20 seconds or more. Partway through, reload the page, open the conversation on a second device, and attach a separate observer client, including straight after a simulated network drop. Each should show the response so far almost immediately, then continue live, with text that matches the original exactly.
How buffering, batching, and rewrites perform on the two tests #
Take a model that emits 120 token events a second over a 20-second response, and a viewer on a slow connection that can take only 40. The surplus tokens have to go somewhere. Here's where each approach puts them, and how each performs on the two tests:
Each approach moves the extra tokens somewhere that costs you: memory and latency, a growing record count, or a write bill. Only rewrites keep one usable response, and they pay for it at every flush.
Message appends: an alternative way to deliver AI responses under load #
Buffering, batching, and record rewrites leave you choosing between a fragmented response and a full write at every flush. Ably message appends remove that tradeoff: your agent creates one message for the response and appends each new piece of text to it. Ably handles ordering, rollup, fan-out, and recovery, so the response stays one message however delivery changes.
Unlike a chunk log, clients get one growing message. Unlike a rewrite, your agent sends only the new text, not the whole response.
How Ably handles one streamed AI response under load
A streamed AI response needs two things from Ably: live delivery while the model generates, and catch-up whenever a viewer joins or reconnects. For live delivery, your agent creates one message and appends to it, and Ably orders, combines, and fans out every append. For catch-up, Ably gives each viewer the response so far as one message.
Here's how that plays out in the five moments a response goes through under load, from the first token to the final record:
Taken together, these moments mean message appends pass both tests, and the response stays one message throughout:
- Slow consumer: gets at most 25 updates a second for each response. If its connection still exceeds itsoutbound rate limit , Ably signals a continuity loss (error 80020 ), and the viewer fetches the response so far.
- Late joiner: gets the response so far as one full update, then live appends.
Handling backpressure outside the delivery layer
Message appends handle backpressure at the delivery layer. To handle it end to end in AI streaming, your application still manages the other three points where it builds:
- Inference: limit concurrent model calls, reject work when capacity runs out, shed load, and cancel model calls through your own request path.
- Worker queues: cap queue size, and expire requests that wait too long.
- Clients: render updates by applying full updates and appends to each viewer's copy, and decide what happens when a viewer falls too far behind: catch up, refetch, or disconnect.
Your application also owns anything your agent does outside the stream: idempotency keys for side effects such as tool calls and payments, and saving completed responses to your database.
Buffering, batching, rewrites, and appends compared #
Side by side, buffering and batching cope with load by splitting the response into records, and record rewrites keep it whole by paying for a full write at every flush. Of the four approaches, only message appends change how often tokens move without changing how many messages a response becomes.
To stream AI responses with message appends, start with the API, then the token streaming pattern for the agent side. Publish over a realtime connection so appends stay in order, and enable the channel rule for message annotations, updates, deletes, and appends first. For recovery behavior, see reconnection and recovery, and read how Ably engineered message appends.
For buffering and batching, the comparison assumes chunks and batches are stored, as they need to be for a late joiner to see anything. Here's the full comparison of one AI response under load:
FAQs #
Doesn't TCP flow control already handle backpressure in AI streaming?
Only one connection at a time. SSE, WebSockets, and HTTP streaming all run over TCP, or QUIC with HTTP/3, and their flow control stops a sender from overwhelming a single receiver, so a slow viewer's data piles up in the server's buffer for that connection instead. Flow control doesn't decide what happens to that backlog, help a viewer who joins mid-response, or coordinate one response across many viewers, so your delivery layer still has to.
Does backpressure matter if only one person watches each AI response?
Yes, though it shows up differently. A single viewer on a congested mobile network or in a background tab can still fall behind the model, and a page refresh or reconnect turns them into a late joiner of their own response. More viewers multiply the cost, but one slow or returning viewer is enough to expose how your delivery layer stores the response.
How can you tell when an AI streaming system is under backpressure?
Watch the gap between how fast the model generates and how fast each viewer receives. Warning signs are per-viewer lag that grows during a response, queue depth or memory that keeps rising on your servers, and final tokens arriving well after the model finishes. On Ably, a connection that exceeds its outbound rate limit receives a continuity-loss signal (error 80020), which your client can log and alert on.
Should one slow viewer throttle the model in AI streaming?
Usually not. When several people watch the same response, slowing generation for one lagging viewer delays everyone else and keeps the model call running longer. Let that viewer catch up from the current state of the response instead, and save throttling the model for real capacity limits, such as too many concurrent requests.