If you self-host large language models, you have probably settled into one of two camps. You use vLLM or SGLang because they are easy to run, or you use TensorRT-LLM because it squeezes the most out of NVIDIA hardware and you are willing to pay for that in setup pain.
TokenSpeed wants to end that trade-off. It is an open-source (MIT) inference engine from the LightSeek Foundation that targets TensorRT-LLM-level performance with vLLM-level usability, and it is tuned specifically for agentic workloads like coding assistants.
In this post we will cover what it is, why "agentic" changes the problem, how to get a server running in a few minutes, and whether it deserves a spot in your stack.
Most inference benchmarks look like chat: a short prompt, a short answer, done. Coding agents look nothing like that.
A typical agent session has:
So the goal is not just "maximum throughput." It is maximum throughput per GPU while every user still gets fast generation. That framing drives most of TokenSpeed's design choices.
You do not need to understand the internals to use it, but four pieces are worth knowing about:
Modeling layer. Instead of hand-writing tensor-parallel communication code for every model, developers annotate where data should live at module boundaries, and a small static compiler generates the collective operations. Less custom parallelism code means fewer bugs when adding new models.
Scheduler. The control plane is written in C++ as a finite-state machine. Request lifecycle and KV cache ownership are modeled explicitly, and the type system catches unsafe KV cache reuse at compile time instead of at 3 a.m. in production. The execution side stays in Python so it is still easy to hack on.
Kernels. Kernels live in their own pluggable package with a registry, so NVIDIA and AMD backends can sit side by side. Their MLA (Multi-head Latent Attention) kernel for Blackwell is fast enough that vLLM has adopted it.
Entrypoint. Requests come in through an OpenAI-compatible server built on their SMG project, keeping CPU-side overhead low.
The project ships a runner image with everything preinstalled:
docker pull lightseekorg/tokenspeed-runner:latest
docker run -itd \
--shm-size 32g \
--gpus all \
--ipc=host \
--network=host \
-v ~/.cache:/home/runner/.cache \
--name tokenspeed \
lightseekorg/tokenspeed-runner:latest \
/bin/bash
Mounting your cache directory means model weights survive container restarts.
On a CUDA 13 machine:
python -m pip install --upgrade tokenspeed \
--extra-index-url https://lightseek.org/whl/nightly
Nightlies are versioned by date, so you can pin one for reproducibility:
python -m pip install "tokenspeed==0.1.0.post20260930" \
--extra-index-url https://lightseek.org/whl/nightly
AMD users should follow the ROCm instructions in the getting started guide, since you need to install ROCm PyTorch first and use a separate wheel index.
Check that everything is in place:
tokenspeed env
tokenspeed serve --help
GPT-OSS 20B fits on a single GPU and is a good smoke test:
tokenspeed serve openai/gpt-oss-20b \
--served-model-name gpt-oss-20b \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 1
Because the server speaks the OpenAI API, your existing client code works with only a base_url change:
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
model="gpt-oss-20b",
messages=[{"role": "user", "content": "Explain KV caching in two sentences."}],
max_tokens=200,
)
print(response.choices[0].message.content)
Or with curl:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-oss-20b",
"messages": [{"role": "user", "content": "Hello!"}],
"max_tokens": 100
}'
That is the whole loop. Anything that already talks to OpenAI, including most agent frameworks, can point at this server.
For large MoE models the flags will look familiar if you have used vLLM. Here is the shape of a Kimi K2.5 launch on four Blackwell GPUs:
tokenspeed serve nvidia/Kimi-K2.5-NVFP4 \
--served-model-name kimi-k2.5 \
--trust-remote-code \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--quantization nvfp4 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--reasoning-parser kimi_k25 \
--tool-call-parser kimik2 \
--host 0.0.0.0 --port 8000
The model recipes page has tested launch commands for DeepSeek V4, Qwen3 and Qwen3.8, GLM 5.x, MiniMax M3, Kimi K3, Nemotron-3 Super, and more. Start from a recipe rather than tuning flags from scratch.
The team's headline numbers, all self-reported:
These are vendor benchmarks, so treat them as a reason to test, not as a guarantee. The good news is the benchmark harness is in the repo under test/agentic_benchmark, so you can reproduce it on your own traffic.
Yes, if: