{"slug": "tokenspeed-a-new-llm-inference-engine-built-for-coding-agents", "title": "TokenSpeed: A New LLM Inference Engine Built for Coding Agents.", "summary": "The LightSeek Foundation has released TokenSpeed, an open-source (MIT) LLM inference engine that targets TensorRT-LLM-level performance with vLLM-level usability, tuned specifically for agentic workloads such as coding assistants. The engine pairs a C++ finite-state-machine scheduler with a Python execution layer, a pluggable kernel registry whose Blackwell MLA kernel has been adopted by vLLM, and an OpenAI-compatible server, and it ships as a preinstalled runner image with pip nightlies for CUDA 13 and ROCm support.", "body_md": "If you self-host large language models, you have probably settled into one of two camps. You use vLLM or SGLang because they are easy to run, or you use TensorRT-LLM because it squeezes the most out of NVIDIA hardware and you are willing to pay for that in setup pain.\n\nTokenSpeed wants to end that trade-off. It is an open-source (MIT) inference engine from the LightSeek Foundation that targets TensorRT-LLM-level performance with vLLM-level usability, and it is tuned specifically for agentic workloads like coding assistants.\n\nIn this post we will cover what it is, why \"agentic\" changes the problem, how to get a server running in a few minutes, and whether it deserves a spot in your stack.\n\nMost inference benchmarks look like chat: a short prompt, a short answer, done. Coding agents look nothing like that.\n\nA typical agent session has:\n\nSo the goal is not just \"maximum throughput.\" It is maximum throughput per GPU *while every user still gets fast generation*. That framing drives most of TokenSpeed's design choices.\n\nYou do not need to understand the internals to use it, but four pieces are worth knowing about:\n\n**Modeling layer.** Instead of hand-writing tensor-parallel communication code for every model, developers annotate where data should live at module boundaries, and a small static compiler generates the collective operations. Less custom parallelism code means fewer bugs when adding new models.\n\n**Scheduler.** The control plane is written in C++ as a finite-state machine. Request lifecycle and KV cache ownership are modeled explicitly, and the type system catches unsafe KV cache reuse at compile time instead of at 3 a.m. in production. The execution side stays in Python so it is still easy to hack on.\n\n**Kernels.** Kernels live in their own pluggable package with a registry, so NVIDIA and AMD backends can sit side by side. Their MLA (Multi-head Latent Attention) kernel for Blackwell is fast enough that vLLM has adopted it.\n\n**Entrypoint.** Requests come in through an OpenAI-compatible server built on their SMG project, keeping CPU-side overhead low.\n\nThe project ships a runner image with everything preinstalled:\n\n```\ndocker pull lightseekorg/tokenspeed-runner:latest\n\ndocker run -itd \\\n  --shm-size 32g \\\n  --gpus all \\\n  --ipc=host \\\n  --network=host \\\n  -v ~/.cache:/home/runner/.cache \\\n  --name tokenspeed \\\n  lightseekorg/tokenspeed-runner:latest \\\n  /bin/bash\n```\n\nMounting your cache directory means model weights survive container restarts.\n\nOn a CUDA 13 machine:\n\n```\npython -m pip install --upgrade tokenspeed \\\n  --extra-index-url https://lightseek.org/whl/nightly\n```\n\nNightlies are versioned by date, so you can pin one for reproducibility:\n\n```\npython -m pip install \"tokenspeed==0.1.0.post20260930\" \\\n  --extra-index-url https://lightseek.org/whl/nightly\n```\n\nAMD users should follow the ROCm instructions in the [getting started guide](https://lightseek.org/tokenspeed/guides/getting-started), since you need to install ROCm PyTorch first and use a separate wheel index.\n\nCheck that everything is in place:\n\n```\ntokenspeed env\ntokenspeed serve --help\n```\n\nGPT-OSS 20B fits on a single GPU and is a good smoke test:\n\n```\ntokenspeed serve openai/gpt-oss-20b \\\n  --served-model-name gpt-oss-20b \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --tensor-parallel-size 1\n```\n\nBecause the server speaks the OpenAI API, your existing client code works with only a `base_url` change:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\n\nresponse = client.chat.completions.create(\n    model=\"gpt-oss-20b\",\n    messages=[{\"role\": \"user\", \"content\": \"Explain KV caching in two sentences.\"}],\n    max_tokens=200,\n)\nprint(response.choices[0].message.content)\n```\n\nOr with curl:\n\n```\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"gpt-oss-20b\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"max_tokens\": 100\n  }'\n```\n\nThat is the whole loop. Anything that already talks to OpenAI, including most agent frameworks, can point at this server.\n\nFor large MoE models the flags will look familiar if you have used vLLM. Here is the shape of a Kimi K2.5 launch on four Blackwell GPUs:\n\n```\ntokenspeed serve nvidia/Kimi-K2.5-NVFP4 \\\n  --served-model-name kimi-k2.5 \\\n  --trust-remote-code \\\n  --max-model-len 262144 \\\n  --kv-cache-dtype fp8 \\\n  --quantization nvfp4 \\\n  --tensor-parallel-size 4 \\\n  --enable-expert-parallel \\\n  --reasoning-parser kimi_k25 \\\n  --tool-call-parser kimik2 \\\n  --host 0.0.0.0 --port 8000\n```\n\nThe [model recipes page](https://lightseek.org/tokenspeed/recipes/models) has tested launch commands for DeepSeek V4, Qwen3 and Qwen3.8, GLM 5.x, MiniMax M3, Kimi K3, Nemotron-3 Super, and more. Start from a recipe rather than tuning flags from scratch.\n\nThe team's headline numbers, all self-reported:\n\nThese are vendor benchmarks, so treat them as a reason to test, not as a guarantee. The good news is the benchmark harness is in the repo under `test/agentic_benchmark`, so you can reproduce it on your own traffic.\n\n**Yes, if:**", "url": "https://wpnews.pro/news/tokenspeed-a-new-llm-inference-engine-built-for-coding-agents", "canonical_source": "https://dev.to/arshtechpro/tokenspeed-a-new-llm-inference-engine-built-for-coding-agents-34jc", "published_at": "2026-10-15 22:42:25+00:00", "updated_at": "2026-10-15 22:58:43.238374+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "developer-tools", "mlops"], "entities": ["TokenSpeed", "LightSeek Foundation", "vLLM", "SGLang", "TensorRT-LLM", "NVIDIA", "AMD", "GPT-OSS 20B"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tokenspeed-a-new-llm-inference-engine-built-for-coding-agents", "markdown": "https://wpnews.pro/news/tokenspeed-a-new-llm-inference-engine-built-for-coding-agents.md", "text": "https://wpnews.pro/news/tokenspeed-a-new-llm-inference-engine-built-for-coding-agents.txt", "jsonld": "https://wpnews.pro/news/tokenspeed-a-new-llm-inference-engine-built-for-coding-agents.jsonld"}}