cd /news/large-language-models/how-to-read-llm-latency-numbers-ttft… · home › topics › large-language-models › article
[ARTICLE · art-174124] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=· neutral

How to read LLM latency numbers: TTFT, sample count, and why one measurement is worthless

A developer measured time-to-first-token (TTFT) across several LLM APIs and found that run-to-run variance within a single model often exceeds the gap between competing models, making single-sample latency comparisons unreliable. In one window, kimi-k2.8-preview ranged from 1.671 s to 4.789 s across three calls placed about a second apart, while qwen3.8-max and minimax-m3 had medians 12% apart but internal spreads of 2.6x and 1.9x. Earlier windows for deepseek-v4-pro showed a roughly 22x degradation that later fully recovered, and glm-4.7-flash returned HTTP 429 on all three probes, which the developer reads as a capacity or quota signal rather than a latency figure.

recommendation

Probe each model at least three times with stream: true and max_tokens: 16, then report the median and range instead of a single TTFT number.

more in playbook: Evals and testing for LLM apps
by read3 min views1 publishedNov 22, 2026

If you have ever picked a model because it "felt faster", you have probably been reading noise.

Across four measurement windows against the same API, one pattern kept showing up: the spread inside three consecutive calls is larger than the difference between most adjacent models in a latency leaderboard.

Below is the fourth window, measured 2026-11-22 03:01 UTC. Three consecutive samples per model, about 1.5 s apart, stream: true, max_tokens: 16, prompt "Reply with exactly: OK". The number is time to first token, in seconds.

model samples median spread
deepseek-v4-flash 0.237 / 0.198 / 0.322 0.237 1.6x
deepseek-v4-pro 0.254 / 0.251 / 0.360 0.254 1.4x
qwen3.8-max 3.068 / 1.766 / 1.187 1.766 2.6x
minimax-m3 1.363 / 2.556 / 1.977 1.977 1.9x
kimi-k2.8-preview 3.735 / 4.789 / 1.671 3.735 2.9x
glm-4.7-flash 429 / 429 / 429 - -

Three findings, in order of how much they should change your reading.

qwen3.8-max and minimax-m3 have medians of 1.77 s and 1.98 s, a 12% difference, and internal spreads of 2.6x and 1.9x. "Which of these two is faster" is not answerable from this data. The noise inside one model is an order of magnitude larger than the gap between the two.

kimi-k2.8-preview is the clearest case: 1.671 s to 4.789 s across three calls placed one second apart. A single probe of that model could honestly report "fast" or "slow" and be wrong about half the time.

A TTFT figure therefore needs at least three samples plus a reported median and range. One number is not a measurement, it is a draw from a distribution.

Earlier windows for deepseek-v4-pro, same probe:

timepoint median TTFT
T1 0.20 s
T2 4.99 s
T3 4.39 s
T4 0.254 s

Two consecutive windows showed roughly a 22x degradation, and then it recovered fully.

I reported T2 and T3 as a 22x swing. With T4 in hand the honest description is "a transient episode that recovered". Anyone reading only T2 and T3 would have concluded the model was persistently degraded, and would have been wrong.

That generalises: one bad reading is not evidence of unreliability, and two consecutive bad readings are still not a trend if you stop measuring after them.

glm-4.7-flash returned HTTP 429 on all three probes, one and a half seconds apart, with no other traffic from the same key. That is not a latency number. It is a capacity or quota finding wearing the costume of a missing row. It also happened to be one of the five models the feed lists as free, and a free model is the one most likely to be saturated.

The reproduction is four lines:

POST /v1/chat/completions
{"model": "<id>", "stream": true, "max_tokens": 16,
 "messages": [{"role": "user", "content": "Reply with exactly: OK"}]}

Measure the time to the first SSE chunk that is not [DONE]. Repeat at least three times. Report the median and the range.

── more in #large-language-models 4 stories · sorted by recency
── more on @deepseek-v4-flash 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-read-llm-late…] indexed:0 read:3min 2026-11-22 · —