We serve GLM-5.2 to teams building agents. Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: the backend you pick changes time-to-first-token by 8×, and one of them wasn’t really streaming at all.
The setup #
Two backends for the identical model. A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting. Both accept a stream: true
request and return an SSE stream. On paper, interchangeable.
What we measured #
We fire 40 cache-busted requests at each, streaming, and record per request: time-to-first-token, total time, and — the important one — whether tokens actually arrive incrementally or all at once at the end. Numbers (p50, 400-token outputs):
| Dedicated host | Cloud managed endpoint | |
|---|---|---|
| TTFT | 1.2s | 10.0s |
| Total | 5.2s | 10.4s |
| Output tok/s per stream | 77 | 38 |
| Actually streamed? | yes (167 chunks) | no — 1 burst at the end |
The cheap endpoint generates the entire response, sits on it for ~10 seconds emitting nothing, then dumps all the tokens at once. It’s stream: true
-compliant to the letter — the bytes do arrive as SSE — but there’s no incremental delivery. For a chat UI or an agent loop, that’s a dead 10-second every turn. The “streaming” is cosmetic.
We assumed it was distance #
It’s a cloud region far from us; surely that’s the latency. So we moved it to a closer region. Result: ~18% faster, still buffered. Region wasn’t the cause — the endpoint buffers generation server-side regardless. And we confirmed it wasn’t our own gateway by running the dedicated host through the exact same path: it streamed 167 chunks cleanly. The buffering belongs to the provider.
Then the twist: caching changes everything #
Cache-busted tests are worst-case. Real agent traffic is heavily cached — long stable system prompts and tool definitions on every turn, only the tail changing. So we re-ran with a warm cached prefix, and the cheap endpoint’s TTFT dropped from 10s to 2.7s, and it mostly stopped buffering. The pathological number was an artifact of cold, uncached requests that no production agent actually sends. Under realistic cached load the gap narrows a lot — though the dedicated host still won ~1.6× on every axis.
That reversal is the real lesson. If we’d benchmarked cache-busted and called it a day, we’d have written the endpoint off as unusably slow. If we’d trusted the provider’s spec sheet, we’d never have seen the buffering at all.
The takeaway #
“OpenAI-compatible” tells you the request shape, nothing about behavior. Two backends serving the identical model differed 8× on the metric users feel, one faked streaming, region didn’t fix it, and cache shape flipped the whole conclusion.
Benchmark backends in your real traffic shape — cached the way you actually run — not with a cache-busted synthetic loop, and not from the provider’s spec sheet. The number that matters is per-stream effective throughput on cached prompts, measured through your own gateway. We define it as output tokens divided by (settle time − first-token time), p50, with cache-fast responses excluded — the speed a single generation actually feels, not a summed-stream aggregate that flatters you under load.
We kept the dedicated host as primary and the cloud endpoint as a failover — it has a much larger context window the other can’t match, and under warm cache it’s perfectly usable. But we only knew any of this because we benchmark every backend before it serves a request — and increasingly, on a schedule, because models drift and providers change serving stacks without telling anyone. If “did my provider quietly break streaming or tool-calls last night” is a question you’ve never been able to answer, that’s the thing we’re building.