The same LLM is 8x slower to first token depending on who serves it

A company serving the same open-weight LLM (GLM-5.2) found that time-to-first-token varied by 8× between a dedicated inference host and a major cloud provider's managed endpoint, with the cloud endpoint buffering all tokens and delivering them in a single burst despite claiming to stream. The gap narrowed under realistic cached traffic, but the dedicated host still outperformed by 1.6×. The findings highlight that "OpenAI-compatible" APIs can differ dramatically in behavior, and benchmarks must use real traffic patterns rather than cache-busted synthetic tests.

We serve GLM-5.2 to teams building agents. Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: the backend you pick changes time-to-first-token by 8×, and one of them wasn’t really streaming at all. The setup Two backends for the identical model. A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting. Both accept a stream: true request and return an SSE stream. On paper, interchangeable. What we measured We fire 40 cache-busted requests at each, streaming, and record per request: time-to-first-token, total time, and — the important one — whether tokens actually arrive incrementally or all at once at the end. Numbers p50, 400-token outputs : | Dedicated host | Cloud managed endpoint | | |---|---|---| | TTFT | 1.2s | 10.0s | | Total | 5.2s | 10.4s | | Output tok/s per stream | 77 | 38 | | Actually streamed? | yes 167 chunks | no — 1 burst at the end | The cheap endpoint generates the entire response, sits on it for ~10 seconds emitting nothing, then dumps all the tokens at once. It’s stream: true -compliant to the letter — the bytes do arrive as SSE — but there’s no incremental delivery. For a chat UI or an agent loop, that’s a dead 10-second pause every turn. The “streaming” is cosmetic. We assumed it was distance It’s a cloud region far from us; surely that’s the latency. So we moved it to a closer region. Result: ~18% faster, still buffered. Region wasn’t the cause — the endpoint buffers generation server-side regardless. And we confirmed it wasn’t our own gateway by running the dedicated host through the exact same path: it streamed 167 chunks cleanly. The buffering belongs to the provider. Then the twist: caching changes everything Cache-busted tests are worst-case. Real agent traffic is heavily cached — long stable system prompts and tool definitions on every turn, only the tail changing. So we re-ran with a warm cached prefix, and the cheap endpoint’s TTFT dropped from 10s to 2.7s , and it mostly stopped buffering. The pathological number was an artifact of cold, uncached requests that no production agent actually sends. Under realistic cached load the gap narrows a lot — though the dedicated host still won ~1.6× on every axis. That reversal is the real lesson. If we’d benchmarked cache-busted and called it a day, we’d have written the endpoint off as unusably slow. If we’d trusted the provider’s spec sheet, we’d never have seen the buffering at all. The takeaway “OpenAI-compatible” tells you the request shape, nothing about behavior. Two backends serving the identical model differed 8× on the metric users feel, one faked streaming, region didn’t fix it, and cache shape flipped the whole conclusion. Benchmark backends in your real traffic shape — cached the way you actually run — not with a cache-busted synthetic loop, and not from the provider’s spec sheet. The number that matters is per-stream effective throughput on cached prompts, measured through your own gateway. We define it as output tokens divided by settle time − first-token time , p50, with cache-fast responses excluded — the speed a single generation actually feels, not a summed-stream aggregate that flatters you under load. We kept the dedicated host as primary and the cloud endpoint as a failover — it has a much larger context window the other can’t match, and under warm cache it’s perfectly usable. But we only knew any of this because we benchmark every backend before it serves a request — and increasingly, on a schedule, because models drift and providers change serving stacks without telling anyone. If “did my provider quietly break streaming or tool-calls last night” is a question you’ve never been able to answer, that’s the thing we’re building.