# The same LLM is 8x slower to first token depending on who serves it > Source: > Published: 2026-07-16 20:20:06+00:00 We serve GLM-5.2 to teams building agents. Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: **the backend you pick changes time-to-first-token by 8×, and one of them wasn’t really streaming at all.** ## The setup Two backends for the identical model. A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting. Both accept a `stream: true` request and return an SSE stream. On paper, interchangeable. ## What we measured We fire 40 cache-busted requests at each, streaming, and record per request: time-to-first-token, total time, and — the important one — whether tokens actually arrive incrementally or all at once at the end. Numbers (p50, 400-token outputs): | Dedicated host | Cloud managed endpoint | | |---|---|---| | TTFT | 1.2s | 10.0s | | Total | 5.2s | 10.4s | | Output tok/s per stream | 77 | 38 | | Actually streamed? | yes (167 chunks) | no — 1 burst at the end | The cheap endpoint generates the entire response, sits on it for ~10 seconds emitting nothing, then dumps all the tokens at once. It’s `stream: true` -compliant to the letter — the bytes do arrive as SSE — but there’s no incremental delivery. For a chat UI or an agent loop, that’s a dead 10-second pause every turn. The “streaming” is cosmetic. ## We assumed it was distance It’s a cloud region far from us; surely that’s the latency. So we moved it to a closer region. Result: ~18% faster, still buffered. Region wasn’t the cause — the endpoint buffers generation server-side regardless. And we confirmed it wasn’t our own gateway by running the dedicated host through the exact same path: it streamed 167 chunks cleanly. The buffering belongs to the provider. ## Then the twist: caching changes everything Cache-busted tests are worst-case. Real agent traffic is heavily cached — long stable system prompts and tool definitions on every turn, only the tail changing. So we re-ran with a warm cached prefix, and the cheap endpoint’s TTFT dropped from 10s to **2.7s**, and it mostly stopped buffering. The pathological number was an artifact of cold, uncached requests that no production agent actually sends. Under realistic cached load the gap narrows a lot — though the dedicated host still won ~1.6× on every axis. That reversal is the real lesson. If we’d benchmarked cache-busted and called it a day, we’d have written the endpoint off as unusably slow. If we’d trusted the provider’s spec sheet, we’d never have seen the buffering at all. ## The takeaway “OpenAI-compatible” tells you the request shape, nothing about behavior. Two backends serving the identical model differed 8× on the metric users feel, one faked streaming, region didn’t fix it, and cache shape flipped the whole conclusion. **Benchmark backends in your real traffic shape — cached the way you actually run — not with a cache-busted synthetic loop, and not from the provider’s spec sheet.** The number that matters is per-stream effective throughput on cached prompts, measured through your own gateway. We define it as output tokens divided by (settle time − first-token time), p50, with cache-fast responses excluded — the speed a single generation actually feels, not a summed-stream aggregate that flatters you under load. We kept the dedicated host as primary and the cloud endpoint as a failover — it has a much larger context window the other can’t match, and under warm cache it’s perfectly usable. But we only knew any of this because we benchmark every backend before it serves a request — and increasingly, on a schedule, because models drift and providers change serving stacks without telling anyone. If *“did my provider quietly break streaming or tool-calls last night”* is a question you’ve never been able to answer, that’s the thing we’re building.