HAProxy's 200 ms LLM token delay is a smart default. Here's when to change it.

An HAProxy engineer investigated a benchmark showing the proxy adds roughly 206 ms of latency to LLM token streams and found the cause is not application-level buffering but the kernel's MSG_MORE flag, which HAProxy sets on client-facing writes while a response body is still being received. The delay is a documented, intentional default that batches small writes into full packets for bulk HTTP traffic, and it can be reversed with a single line of configuration for token-streaming workloads.

NOTE: I work at HAProxy, so I'm probably a bit biased. Still, this is an accurate look at the situation. A fun little deep dive, actually. Regardless, this is my own opinion/research, not official from the company. A recent benchmark post https://dev.to/remdore/nginx-streams-your-tokens-fine-haproxy-holds-them-for-206ms-10p2 made the rounds with a table that seems bad for HAProxy. Four proxies sat in front of a scripted LLM emitter sending one tiny frame every 50 milliseconds. Through nginx, Caddy, and Traefik, tokens arrived 2 to 3 milliseconds after the proxy got them, one per read, indistinguishable from no proxy at all. Through HAProxy, the first token showed up 206 milliseconds late and the rest arrived in bursts of five with no gap between them. The post's verdict: HAProxy buffers your tokens. That sounded unusual for such a performant tool, so I decided to dig into it a bit. Here is what I found: nothing in HAProxy is collecting tokens in a bucket . The delay comes from HAProxy telling the kernel to wait for more data, on purpose, documented almost to the millisecond, and reversible with one line of config. And the answer to "why is it on by default" is that for most of the traffic HAProxy carries, the default is the faster choice. The background is Nagle's algorithm https://en.wikipedia.org/wiki/Nagle's algorithm , named for John Nagle and described in RFC 896 back in 1984: a sender holding small amounts of data should wait for a full packet's worth, or for the receiver to acknowledge the previous packet, before putting another tiny packet on the wire. The reasoning is arithmetic. Without batching, a 60-byte write pays a header-sized envelope, and those headers can double the bytes transmitted, or worse as frames shrink. Nagle isn't the actor here, though . HAProxy already sets TCP NODELAY on its TCP sockets. What holds the tokens is a second, explicit mechanism: while a response body is still being received, HAProxy tags every client-facing write with the kernel's MSG MORE flag, which tells the kernel "more data is coming, hold this." In the 3.4 source the article tested, the H1 mux sets the tag on every in-flight body write src/mux h1.c , via CO SFL MSG MORE and clears it when the body ends. The configuration manual https://docs.haproxy.org/2.2/configuration.html 4.2-option%20http-no-delay describes the effect exactly: the system "waits for enough data to be available in order to only send full packets. Typical delays are around 200 ms per round trip." A token stream never fills a packet. Sixty bytes every 50ms, with a round trip sitting between each one, so the corked writes sit in the kernel until the tag clears or a timer cycle releases them. The tokens leave in batches, late by a few cycles, and they reach the client mashed together: five frames, zero gap, 206 ms. That matches the benchmark numbers. First, this is not application buffering. HAProxy forwards the response body as it arrives. The waiting happens in the kernel's send path, not in a proxy buffer. Second, the benchmark author says this himself: frames-per-read is a property of the stream shape, not a permanent fact about the proxy. Same proxy, 1.1 KB frames instead of 60-byte ones, and the delay drops to 53 ms with near-zero coalescing. The batching only bites when frames are small relative to a packet, which is, unfortunately, the exact shape of an LLM token. For the traffic HAProxy was built to carry, batching is a measurable win, and the costs fall on workloads that never notice. HAProxy's home turf is bulk HTTP at high concurrency: API responses, uploads, downloads, and everything in between, at request rates where packet counts and syscall overhead show up directly on the CPU graph. Every tiny write batched into a bigger packet is a syscall saved, a header not transmitted, and a few microseconds returned to the event loop. The HAProxy troubleshooting guide <https://docs.haproxy.org/2.2/management.html :~:text=definitely%20remains%0Aenabled%20when%20forwarding%20an%20HTTP%20body%20 and%20this%20contributes%20to%20the%20performance%0Aimprovement%20there%20by%20reducing%20the%20number%20of%20packets states the trade plainly: Nagle " definitely remains enabled when forwarding an HTTP body and this contributes to the performance improvement there by reducing the number of packets . " For those workloads the latency cost is invisible, because a download or an API response doesn't care about 40 ms of accumulated batching once per connection; the payload arrives in bulk regardless, and nobody waiting on it can tell the difference. The design also thought about interactive traffic. The batching is skipped automatically in pure TCP mode and in tunnels. WebSockets and CONNECT requests are explicitly documented as unaffected. The default was tuned with a clear picture of what "interactive" meant at the time: a tunnel, an upgraded socket, or someone pushing a protocol through HTTP that the HTTP spec never intended. Which brings us to the core issue. Server-sent events are plain chunked HTTP. There's no upgrade, no tunnel, nothing to trip the existing exemptions. To HAProxy, an SSE response is an ordinary body that happens to dribble in, so every client-facing write carries the hold tag. The benchmark measured 206. The manual's " around 200 ms per round trip " predicted it because its just math. None of this is hidden, and none of it is an accident. The defaults were tuned for the traffic HAProxy has carried for most of its life: bulk responses at high concurrency, workloads where waiting to fill a packet costs nothing and saves real CPU. Token streaming inverts that trade . The frames are tiny, the stream runs for minutes instead of milliseconds, and the wait lands between a model and a user watching the screen. A default shaped for one kind of traffic cannot also be shaped for its opposite, and bending it until it fits both is how you end up with slower defaults for everyone. This is precisely why the adjustment exists. option http-no-delay is the designed escape hatch for workloads whose trade-offs differ from the common case, documented in the same manual as the default itself. The system works the way a configurable system should: fast by default for the many, one line to match the few . The benchmark's real contribution is showing which default fits which traffic, and for token streaming the answer is one line long. backend llm-inference option http-no-delay server inference 10.0.0.5:8000 check One line, in the backend carrying the stream. It tells HAProxy to stop tagging the writes, so the kernel never holds them, and the manual describes the result directly: " all such optimizations will be disabled in order to make the exchanges as fast as possible. " Frames leave when they arrive. This isn't just documentation. The benchmark was reproduced on loopback with the same stream shape the article used, 60-byte chunked SSE events every 50 ms, running a local HAProxy 3.2 after reading the 3.4 source the article tested: | setup | first event | frames per read | |---|---|---| | direct to origin | 1 ms | 1.00 | | through HAProxy, default | 212 ms | 5.00 | | through HAProxy, option http-no-delay | 1 ms | 1.00 | Same shape as the original numbers, and the one line erases the delay completely: first token back to 1 ms, one frame per read, indistinguishable from no proxy. The benchmark author listed " whether option http-no-delay removes it " as a test he'd still like to run. Consider this cell run, with credit to his rig for the design. With Nagle off, every frame goes out as its own write, which means one packet and one syscall per frame instead of one per full segment. On a 50 ms token cadence that's roughly 20 packets a second per stream, which no individual user will ever notice. Across tens of thousands of concurrent streams on a busy edge, however, it becomes a real and measurable bump in CPU consumption and packet overhead, and the overhead compounds anywhere the network path is slow or congested. The manual warns about this too: the option " should never be used by default ," and its cost " may significantly lower performance in high latency environments ," where every small packet now pays the full round trip alone instead of sharing it with its neighbors. So the move is not to paste it into a global defaults section. Scope it to the backends that stream. Let the bulk traffic keep the batching. A rule of thumb that applies: If the payload is the message, turn it off. If the payload is bulk, leave it on. Tokens, chat deltas, live logs, trading ticks, anything where a human or a model is waiting on each frame: those want option http-no-delay . File downloads, API JSON responses, images: keep the default and keep the efficiency. When it's unclear which side the traffic falls on, measure it. The benchmark harness from the original post is a good template, and the author's write-up of the fourteen ways he broke it before trusting it is worth reading regardless of which proxy runs in front of the service. It's fair to ask, because nginx ships the reverse wager. Its tcp nodelay directive defaults to on, so small writes leave immediately. In exchange, nginx buffers more aggressively at the application layer proxy buffering defaults to on . HAProxy does the opposite: it forwards responses almost immediately and asks the kernel to batch the body writes instead. The benchmark shows both bets working. nginx's buffering cost about 50 ms only in a deliberately nasty case: a 328 KB payload pushed to a client that sleeps 200 ms between reads. For prompt clients on small streams, it costs nothing. HAProxy's batching costs 200 ms only for small-frame streams. For everything else, it's free throughput. Neither default is wrong . nginx is friendly out of the box for streaming and pays slightly more overhead on bulk workloads, while HAProxy is leaner on bulk HTTP at scale and needs one line of config for streams. If the product is a stream of tokens, both are available at once: HAProxy's throughput and nginx's latency, for the price of one config line. A default is an opinion about the common case, and HAProxy's defaults are opinions worth having. They favor throughput, low CPU, and safety at scale, and the numbers behind them show up in every large deployment. The 206 ms number is real, it's documented, it's reproducible, and it's a knob, not a wall. If tokens are streaming through HAProxy today, add the line, re-run the benchmark, and take the milliseconds back. Thanks to the original author https://dev.to/remdore for the careful measurement work. This post started as a technical review of it, and the rig's honesty killed findings, one-shot cells, published configs made the review easy. Photo by Ashe Walker https://unsplash.com/@marsupialpudding?utm source=unsplash&utm medium=referral&utm content=creditCopyText on Unsplash https://unsplash.com/photos/a-speed-bump-sign-on-a-street-corner-b2fbsoUzafY?utm source=unsplash&utm medium=referral&utm content=creditCopyText