If you have ever picked a model because it "felt faster", you have probably been reading noise.
Across four measurement windows against the same API, one pattern kept showing up: the spread inside three consecutive calls is larger than the difference between most adjacent models in a latency leaderboard.
Below is the fourth window, measured 2026-11-22 03:01 UTC. Three consecutive samples per model, about 1.5 s apart, stream: true, max_tokens: 16, prompt "Reply with exactly: OK". The number is time to first token, in seconds.
| model | samples | median | spread |
|---|---|---|---|
| deepseek-v4-flash | 0.237 / 0.198 / 0.322 | 0.237 | 1.6x |
| deepseek-v4-pro | 0.254 / 0.251 / 0.360 | 0.254 | 1.4x |
| qwen3.8-max | 3.068 / 1.766 / 1.187 | 1.766 | 2.6x |
| minimax-m3 | 1.363 / 2.556 / 1.977 | 1.977 | 1.9x |
| kimi-k2.8-preview | 3.735 / 4.789 / 1.671 | 3.735 | 2.9x |
| glm-4.7-flash | 429 / 429 / 429 | - | - |
Three findings, in order of how much they should change your reading.
qwen3.8-max and minimax-m3 have medians of 1.77 s and 1.98 s, a 12% difference, and internal spreads of 2.6x and 1.9x. "Which of these two is faster" is not answerable from this data. The noise inside one model is an order of magnitude larger than the gap between the two.
kimi-k2.8-preview is the clearest case: 1.671 s to 4.789 s across three calls placed one second apart. A single probe of that model could honestly report "fast" or "slow" and be wrong about half the time.
A TTFT figure therefore needs at least three samples plus a reported median and range. One number is not a measurement, it is a draw from a distribution.
Earlier windows for deepseek-v4-pro, same probe:
| timepoint | median TTFT |
|---|---|
| T1 | 0.20 s |
| T2 | 4.99 s |
| T3 | 4.39 s |
| T4 | 0.254 s |
Two consecutive windows showed roughly a 22x degradation, and then it recovered fully.
I reported T2 and T3 as a 22x swing. With T4 in hand the honest description is "a transient episode that recovered". Anyone reading only T2 and T3 would have concluded the model was persistently degraded, and would have been wrong.
That generalises: one bad reading is not evidence of unreliability, and two consecutive bad readings are still not a trend if you stop measuring after them.
glm-4.7-flash returned HTTP 429 on all three probes, one and a half seconds apart, with no other traffic from the same key. That is not a latency number. It is a capacity or quota finding wearing the costume of a missing row. It also happened to be one of the five models the feed lists as free, and a free model is the one most likely to be saturated.
The reproduction is four lines:
POST /v1/chat/completions
{"model": "<id>", "stream": true, "max_tokens": 16,
"messages": [{"role": "user", "content": "Reply with exactly: OK"}]}
Measure the time to the first SSE chunk that is not [DONE]. Repeat at least three times. Report the median and the range.