Note: The Show and Tell category is for sharing and discussing projects, showcasing your Spaces, Models, Datasets and more. We value open-source and technical details over promotional content, so focus on sharing the intricate aspects of your work.Which local inference server is fastest for your model? On your machine, it depends, and benchmark posts rarely transfer.
I kept failing to reproduce other people’s tok/s numbers, so I built TokenRace: a small open-source app that sends the same streaming request to whichever local servers you run and times everything on the client, with the same code for all of them.
You pick a server and any model you have, start the test and get TTFT, tok/s, total time and RAM live, then a summary with median, min and max, a chart and a CSV.
It works with Ollama, LM Studio, llama.cpp, vLLM, SGLang, MLX servers and others (15 out of the box, any OpenAI-compatible server can be added), and runs locally with one command: uvx tokenrace
Repo: [GitHub - ernext/tokenrace: Compare local LLM inference servers (Ollama, LM Studio, MLX, llama.cpp, vLLM…) on your own hardware, with the same client-side timing for all of them. · GitHub](https://github.com/ernext/tokenrace)
I also wrote up the method here: [Why your local LLM tok/s numbers don't match mine - DEV Community](https://dev.to/ernext/why-your-local-llm-toks-numbers-dont-match-mine-5dn6)
If you measure a model on NVIDIA, AMD or Linux, I’d love to see the numbers.