TokenRace: compare local LLM servers on your own hardware

A developer released TokenRace, an open-source app that benchmarks local LLM inference servers on a user's own hardware by sending the same streaming request to each server and timing everything client-side. TokenRace reports time to first token (TTFT), tokens per second, total time and RAM live, then summarizes median, min and max results with a chart and CSV export, and supports 15 servers out of the box including Ollama, LM Studio, llama.cpp, vLLM, SGLang and MLX, plus any OpenAI-compatible server. It runs locally with the command uvx tokenrace, and the developer is soliciting results from NVIDIA, AMD and Linux users.

Note: The Show and Tell category is for sharing and discussing projects, showcasing your Spaces, Models, Datasets and more. We value open-source and technical details over promotional content, so focus on sharing the intricate aspects of your work .Which local inference server is fastest for your model? On your machine, it depends, and benchmark posts rarely transfer. I kept failing to reproduce other people’s tok/s numbers, so I built TokenRace: a small open-source app that sends the same streaming request to whichever local servers you run and times everything on the client, with the same code for all of them. You pick a server and any model you have, start the test and get TTFT, tok/s, total time and RAM live, then a summary with median, min and max, a chart and a CSV. It works with Ollama, LM Studio, llama.cpp, vLLM, SGLang, MLX servers and others 15 out of the box, any OpenAI-compatible server can be added , and runs locally with one command: uvx tokenrace Repo: GitHub - ernext/tokenrace: Compare local LLM inference servers Ollama, LM Studio, MLX, llama.cpp, vLLM… on your own hardware, with the same client-side timing for all of them. · GitHub https://github.com/ernext/tokenrace I also wrote up the method here: Why your local LLM tok/s numbers don't match mine - DEV Community https://dev.to/ernext/why-your-local-llm-toks-numbers-dont-match-mine-5dn6 If you measure a model on NVIDIA, AMD or Linux, I’d love to see the numbers.