{"slug": "the-same-llm-is-8x-slower-to-first-token-depending-on-who-serves-it", "title": "The same LLM is 8x slower to first token depending on who serves it", "summary": "A company serving the same open-weight LLM (GLM-5.2) found that time-to-first-token varied by 8× between a dedicated inference host and a major cloud provider's managed endpoint, with the cloud endpoint buffering all tokens and delivering them in a single burst despite claiming to stream. The gap narrowed under realistic cached traffic, but the dedicated host still outperformed by 1.6×. The findings highlight that \"OpenAI-compatible\" APIs can differ dramatically in behavior, and benchmarks must use real traffic patterns rather than cache-busted synthetic tests.", "body_md": "We serve GLM-5.2 to teams building agents. Same open-weight model, same OpenAI-compatible API — but we route it across more than one backend, and while swapping one in we found something worth writing down: **the backend you pick changes time-to-first-token by 8×, and one of them wasn’t really streaming at all.**\n\n## The setup\n\nTwo backends for the identical model. A dedicated inference host, and a major cloud provider’s managed OpenAI-compatible endpoint — cheaper per token, a very large context window, tempting. Both accept a `stream: true`\n\nrequest and return an SSE stream. On paper, interchangeable.\n\n## What we measured\n\nWe fire 40 cache-busted requests at each, streaming, and record per request: time-to-first-token, total time, and — the important one — whether tokens actually arrive incrementally or all at once at the end. Numbers (p50, 400-token outputs):\n\n| Dedicated host | Cloud managed endpoint | |\n|---|---|---|\n| TTFT | 1.2s | 10.0s |\n| Total | 5.2s | 10.4s |\n| Output tok/s per stream | 77 | 38 |\n| Actually streamed? | yes (167 chunks) | no — 1 burst at the end |\n\nThe cheap endpoint generates the entire response, sits on it for ~10 seconds emitting nothing, then dumps all the tokens at once. It’s `stream: true`\n\n-compliant to the letter — the bytes do arrive as SSE — but there’s no incremental delivery. For a chat UI or an agent loop, that’s a dead 10-second pause every turn. The “streaming” is cosmetic.\n\n## We assumed it was distance\n\nIt’s a cloud region far from us; surely that’s the latency. So we moved it to a closer region. Result: ~18% faster, still buffered. Region wasn’t the cause — the endpoint buffers generation server-side regardless. And we confirmed it wasn’t our own gateway by running the dedicated host through the exact same path: it streamed 167 chunks cleanly. The buffering belongs to the provider.\n\n## Then the twist: caching changes everything\n\nCache-busted tests are worst-case. Real agent traffic is heavily cached — long stable system prompts and tool definitions on every turn, only the tail changing. So we re-ran with a warm cached prefix, and the cheap endpoint’s TTFT dropped from 10s to **2.7s**, and it mostly stopped buffering. The pathological number was an artifact of cold, uncached requests that no production agent actually sends. Under realistic cached load the gap narrows a lot — though the dedicated host still won ~1.6× on every axis.\n\nThat reversal is the real lesson. If we’d benchmarked cache-busted and called it a day, we’d have written the endpoint off as unusably slow. If we’d trusted the provider’s spec sheet, we’d never have seen the buffering at all.\n\n## The takeaway\n\n“OpenAI-compatible” tells you the request shape, nothing about behavior. Two backends serving the identical model differed 8× on the metric users feel, one faked streaming, region didn’t fix it, and cache shape flipped the whole conclusion.\n\n**Benchmark backends in your real traffic shape — cached the way you actually run — not with a cache-busted synthetic loop, and not from the provider’s spec sheet.** The number that matters is per-stream effective throughput on cached prompts, measured through your own gateway. We define it as output tokens divided by (settle time − first-token time), p50, with cache-fast responses excluded — the speed a single generation actually feels, not a summed-stream aggregate that flatters you under load.\n\nWe kept the dedicated host as primary and the cloud endpoint as a failover — it has a much larger context window the other can’t match, and under warm cache it’s perfectly usable. But we only knew any of this because we benchmark every backend before it serves a request — and increasingly, on a schedule, because models drift and providers change serving stacks without telling anyone. If *“did my provider quietly break streaming or tool-calls last night”* is a question you’ve never been able to answer, that’s the thing we’re building.", "url": "https://wpnews.pro/news/the-same-llm-is-8x-slower-to-first-token-depending-on-who-serves-it", "canonical_source": "https://dynoyard.app/blog/same-llm-8x-slower-by-backend/", "published_at": "2026-07-16 20:20:06+00:00", "updated_at": "2026-07-16 20:55:27.523851+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["GLM-5.2", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/the-same-llm-is-8x-slower-to-first-token-depending-on-who-serves-it", "markdown": "https://wpnews.pro/news/the-same-llm-is-8x-slower-to-first-token-depending-on-who-serves-it.md", "text": "https://wpnews.pro/news/the-same-llm-is-8x-slower-to-first-token-depending-on-who-serves-it.txt", "jsonld": "https://wpnews.pro/news/the-same-llm-is-8x-slower-to-first-token-depending-on-who-serves-it.jsonld"}}