I test software for a living. So when a vendor calls an AI model "fast," I don't trust the word. I measure it.
Most leaderboards rank how smart a model is. Almost none rank how fast it answers. You pick a model because it scored well, ship it, and then your users sit and wait.
Speed is two different numbers. People mix them up constantly.
Time to first token (TTFT). The wait before the first word appears. You feel this every time a chatbot "thinks" before replying.
Tokens per second (TPS). How fast the model writes once it starts. A token is a chunk of a word.
A model can be great at one and terrible at the other. You need both.
I run an independent tracker called ollamatps.com. It benchmarks 42 Ollama Cloud models. Here is the exact method, because a benchmark you cannot inspect is just a claim.
max_tokens
is capped at 300. It never changes between runs.eval_count / (total_duration - time_to_first_token)
. The startup wait is removed, so TPS measures pure writing speed.Same prompt, same cap, same schedule. That is what makes two models comparable.
Building this was a testing job, not a coding job. Retries on failure. A reliability score per model. A circuit breaker for models that keep failing. If you cannot trust the measurement, the number is noise. That part is the same work I do on any test system.
Bigger is not faster.
The fastest model on the board is one of the smallest: a 30B model at over 200 tokens per second. The model literally named "ultra" is dead last, under 8 tokens per second.
And the wait varies wildly. TTFT ranges from about 0.3 seconds to 23 seconds across the 42 models. Same cloud. Roughly 80x difference in how long you wait for the first word.
If you picked your model on a benchmark score alone, you have no idea which of these you are getting. The first version tracked fewer models and was less robust. I rebuilt the engine this month (v2) to be multi-provider and to test continuously. The live board updates every 10 minutes.
Watch it run: ollamatps.com Anton Gulin is the AI QA Architect, the first person to claim this title on LinkedIn. He builds AI-powered test automation systems where AI agents and human engineers collaborate on quality. Former Apple SDET (Apple.com / Apple Card pre-release testing). Find him at anton.qa or on LinkedIn.