{"slug": "how-to-read-llm-latency-numbers-ttft-sample-count-and-why-one-measurement-is", "title": "How to read LLM latency numbers: TTFT, sample count, and why one measurement is worthless", "summary": "A developer measured time-to-first-token (TTFT) across several LLM APIs and found that run-to-run variance within a single model often exceeds the gap between competing models, making single-sample latency comparisons unreliable. In one window, kimi-k2.8-preview ranged from 1.671 s to 4.789 s across three calls placed about a second apart, while qwen3.8-max and minimax-m3 had medians 12% apart but internal spreads of 2.6x and 1.9x. Earlier windows for deepseek-v4-pro showed a roughly 22x degradation that later fully recovered, and glm-4.7-flash returned HTTP 429 on all three probes, which the developer reads as a capacity or quota signal rather than a latency figure.", "body_md": "If you have ever picked a model because it \"felt faster\", you have probably been reading noise.\n\nAcross four measurement windows against the same API, one pattern kept showing up: **the spread inside three consecutive calls is larger than the difference between most adjacent models in a latency leaderboard.**\n\nBelow is the fourth window, measured 2026-11-22 03:01 UTC. Three consecutive samples per model, about 1.5 s apart, `stream: true`, `max_tokens: 16`, prompt \"Reply with exactly: OK\". The number is time to first token, in seconds.\n\n| model | samples | median | spread | \n|---|---|---|---|\n| deepseek-v4-flash | 0.237 / 0.198 / 0.322 | 0.237 | 1.6x | \n| deepseek-v4-pro | 0.254 / 0.251 / 0.360 | 0.254 | 1.4x | \n| qwen3.8-max | 3.068 / 1.766 / 1.187 | 1.766 | 2.6x | \n| minimax-m3 | 1.363 / 2.556 / 1.977 | 1.977 | 1.9x | \n| kimi-k2.8-preview | 3.735 / 4.789 / 1.671 | 3.735 | 2.9x | \n| glm-4.7-flash | 429 / 429 / 429 | - | - | \n\nThree findings, in order of how much they should change your reading.\n\nqwen3.8-max and minimax-m3 have medians of 1.77 s and 1.98 s, a 12% difference, and internal spreads of 2.6x and 1.9x. \"Which of these two is faster\" is not answerable from this data. The noise inside one model is an order of magnitude larger than the gap between the two.\n\nkimi-k2.8-preview is the clearest case: 1.671 s to 4.789 s across three calls placed one second apart. A single probe of that model could honestly report \"fast\" or \"slow\" and be wrong about half the time.\n\nA TTFT figure therefore needs at least three samples plus a reported median and range. One number is not a measurement, it is a draw from a distribution.\n\nEarlier windows for deepseek-v4-pro, same probe:\n\n| timepoint | median TTFT | \n|---|---|\n| T1 | 0.20 s | \n| T2 | 4.99 s | \n| T3 | 4.39 s | \n| T4 | 0.254 s | \n\nTwo consecutive windows showed roughly a 22x degradation, and then it recovered fully.\n\nI reported T2 and T3 as a 22x swing. With T4 in hand the honest description is \"a transient episode that recovered\". Anyone reading only T2 and T3 would have concluded the model was persistently degraded, and would have been wrong.\n\nThat generalises: one bad reading is not evidence of unreliability, and two consecutive bad readings are still not a trend if you stop measuring after them.\n\nglm-4.7-flash returned HTTP 429 on all three probes, one and a half seconds apart, with no other traffic from the same key. That is not a latency number. It is a capacity or quota finding wearing the costume of a missing row. It also happened to be one of the five models the feed lists as free, and a free model is the one most likely to be saturated.\n\nThe reproduction is four lines:\n\n```\nPOST /v1/chat/completions\n{\"model\": \"<id>\", \"stream\": true, \"max_tokens\": 16,\n \"messages\": [{\"role\": \"user\", \"content\": \"Reply with exactly: OK\"}]}\n```\n\nMeasure the time to the first SSE chunk that is not `[DONE]`. Repeat at least three times. Report the median and the range.", "url": "https://wpnews.pro/news/how-to-read-llm-latency-numbers-ttft-sample-count-and-why-one-measurement-is", "canonical_source": "https://dev.to/aiapikey/how-to-read-llm-latency-numbers-ttft-sample-count-and-why-one-measurement-is-worthless-2p7l", "published_at": "2026-11-22 10:05:27+00:00", "updated_at": "2026-11-22 10:12:19.372125+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["deepseek-v4-flash", "deepseek-v4-pro", "qwen3.8-max", "minimax-m3", "kimi-k2.8-preview", "glm-4.7-flash"], "also_reported_by": [], "takeaway": "Report at least three TTFT samples with median and range, because single-model spread (up to 2.9x for kimi-k2.8-preview) can exceed the gap between models and a lone probe is a draw from a distribution, not a measurement.", "recommendation": {"advice": "Probe each model at least three times with stream: true and max_tokens: 16, then report the median and range instead of a single TTFT number.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/how-to-read-llm-latency-numbers-ttft-sample-count-and-why-one-measurement-is", "markdown": "https://wpnews.pro/news/how-to-read-llm-latency-numbers-ttft-sample-count-and-why-one-measurement-is.md", "text": "https://wpnews.pro/news/how-to-read-llm-latency-numbers-ttft-sample-count-and-why-one-measurement-is.txt", "jsonld": "https://wpnews.pro/news/how-to-read-llm-latency-numbers-ttft-sample-count-and-why-one-measurement-is.jsonld"}}