{"slug": "i-measure-how-fast-42-llms-actually-answer-here-s-the-honest-method", "title": "I measure how fast 42 LLMs actually answer. Here's the honest method.", "summary": "An independent speed tracker, ollamatps.com, benchmarks 42 Ollama Cloud models by measuring time to first token (TTFT) and tokens per second (TPS). The tracker reveals that smaller models can be faster, with a 30B model achieving over 200 TPS while the 'ultra' model lags at under 8 TPS, and TTFT varies from 0.3 to 23 seconds. The project, built by AI QA Architect Anton Gulin, uses a rigorous method with capped max_tokens and retries to ensure reliable comparisons.", "body_md": "I test software for a living. So when a vendor calls an AI model \"fast,\" I don't trust the word. I measure it.\n\nMost leaderboards rank how smart a model is. Almost none rank how fast it answers. You pick a model because it scored well, ship it, and then your users sit and wait.\n\nSpeed is two different numbers. People mix them up constantly.\n\n**Time to first token (TTFT).** The wait before the first word appears. You feel this every time a chatbot \"thinks\" before replying.\n\n**Tokens per second (TPS).** How fast the model writes once it starts. A token is a chunk of a word.\n\nA model can be great at one and terrible at the other. You need both.\n\nI run an independent tracker called [ollamatps.com](https://ollamatps.com). It benchmarks 42 Ollama Cloud models. Here is the exact method, because a benchmark you cannot inspect is just a claim.\n\n`max_tokens`\n\nis capped at 300. It never changes between runs.`eval_count / (total_duration - time_to_first_token)`\n\n. The startup wait is removed, so TPS measures pure writing speed.Same prompt, same cap, same schedule. That is what makes two models comparable.\n\nBuilding this was a testing job, not a coding job. Retries on failure. A reliability score per model. A circuit breaker for models that keep failing. If you cannot trust the measurement, the number is noise. That part is the same work I do on any test system.\n\n**Bigger is not faster.**\n\nThe fastest model on the board is one of the smallest: a 30B model at over 200 tokens per second. The model literally named \"ultra\" is dead last, under 8 tokens per second.\n\nAnd the wait varies wildly. TTFT ranges from about 0.3 seconds to 23 seconds across the 42 models. Same cloud. Roughly 80x difference in how long you wait for the first word.\n\nIf you picked your model on a benchmark score alone, you have no idea which of these you are getting.\n\nThe first version tracked fewer models and was less robust. I rebuilt the engine this month (v2) to be multi-provider and to test continuously. The live board updates every 10 minutes.\n\nWatch it run: [ollamatps.com](https://ollamatps.com)\n\n*Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.*", "url": "https://wpnews.pro/news/i-measure-how-fast-42-llms-actually-answer-here-s-the-honest-method", "canonical_source": "https://dev.to/aiwithanton/i-measure-how-fast-42-llms-actually-answer-heres-the-honest-method-3gap", "published_at": "2026-06-16 05:06:22+00:00", "updated_at": "2026-06-16 05:17:15.471717+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools", "ai-products", "ai-research"], "entities": ["Ollama", "ollamatps.com", "Anton Gulin", "Apple"], "alternates": {"html": "https://wpnews.pro/news/i-measure-how-fast-42-llms-actually-answer-here-s-the-honest-method", "markdown": "https://wpnews.pro/news/i-measure-how-fast-42-llms-actually-answer-here-s-the-honest-method.md", "text": "https://wpnews.pro/news/i-measure-how-fast-42-llms-actually-answer-here-s-the-honest-method.txt", "jsonld": "https://wpnews.pro/news/i-measure-how-fast-42-llms-actually-answer-here-s-the-honest-method.jsonld"}}