Selected checkpoints. Maximum single-GPU inference performance.
NInfer is a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs.
NInfer deliberately supports a closed set of model artifacts instead of acting as a general model runtime:
| Model | Weights | NInfer artifact | Size | SHA-256 |
|---|---|---|---|---|
groupwise-int
qwen3_6_27b.ninfer
7b51600ffd10632b9660f56085efdd9b751d79733ad32036a652234b64bebe7b
Qwen3.6-27B NVFP4nvfp4
qwen3_6_27b_nvfp4.ninfer
bce5f00d066c0f20f1317bf1fdcb458264cf95837c3b1f3fbec163694627893a
Qwen3.8-27Bgroupwise-int
qwen3_8_27b.ninfer
eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e
Qwen3.6-35B-A3Bgroupwise-int
qwen3_6_35b_a3b.ninfer
1fb9ea0b5b8561e49d9604115ec89e5d9f2b6f6434e32c37c57fffd480a325d2
The two Qwen3.6-27B weight profiles bind to the registered qwen3_6_27b
target; the version-2
artifact identity selects the profile without a separate runtime flag. Qwen3.8-27B is separately
registered as qwen3_8_27b
and shares the 27B execution package while using W8 token-embedding and
full-output-head weights. The nvfp4
profile uses W4A4 Tensor Core MMA for prefill and A16 NVFP4 kernels for decode. All three 27B artifacts retain the same Text, Vision, MTP, prefix-reuse, CLI, and serving routes.
The published measurements currently cover the three Qwen3.6 artifact profiles. Qwen3.8-27B is supported by current NInfer builds but is not yet included in the benchmark campaign.
Saturated decode was measured on an RTX 5090 with INT8 group-64 KV cache, CUDA Graphs, MTP3, and one 8,192-token generation per active request. The values below are aggregate committed decode throughput from complete one-second intervals in which the actual decode batch remained equal to the configured concurrency. Each profile should be read independently.
| Model profile | C=1 | C=2 | C=4 | C=8 | C8 / C1 |
|---|---|---|---|---|---|
Qwen3.6-27B groupwise-int |
|||||
| 185.8 tok/s | 247.0 tok/s | 309.5 tok/s | 535.0 tok/s | 2.88× | |
Qwen3.6-27B nvfp4 |
|||||
| 202.4 tok/s | 399.7 tok/s | 699.7 tok/s | 1,146.9 tok/s | 5.67× | |
Qwen3.6-35B-A3B groupwise-int |
|||||
| 593.0 tok/s | 877.7 tok/s | 1,166.0 tok/s | 1,313.8 tok/s | 2.22× |
At C=8, Qwen3.6-35B-A3B reaches 1,313.8 aggregate decode tok/s. The 27B NVFP4 profile reaches 1,146.9 tok/s and 5.67× its C=1 throughput.
The single-request corpus was measured on the same GPU with INT8 group-64 KV cache, CUDA Graphs, and a 1,024-token prefill chunk. Each reported fixture uses five fixed seeds after server warm-up. The two measured targets are reported independently and are not cross-target comparisons. The two 27B weight profiles are reported separately. Requests were submitted serially to a persistent server.
Qwen3.6-35B-A3B
- MTP0 at a 7,680-token prompt: 15,544.3 prefill tok/s and271.1 decode tok/s. - MTP0 at a 260,096-token prompt: 5,157.1 prefill tok/s and188.2 decode tok/s. - MTP3 long reasoning: 620.3–726.2 decode tok/s with72.7–82.8% acceptance. - MTP3 structured output: 770.9 decode tok/s,** 89.1% acceptance**, and** 3.67 tokens/round**.
Qwen3.6-27B ( groupwise-int)
- MTP0 at a 7,680-token prompt: 3,218.1 prefill tok/s and77.6 decode tok/s. - MTP0 at a 260,096-token prompt: 1,614.8 prefill tok/s and54.8 decode tok/s. - MTP3 long reasoning: 161.9–175.4 decode tok/s with73.4–78.8% acceptance. - MTP3 structured output: 193.0 decode tok/s,** 88.7% acceptance**, and** 3.66 tokens/round**.
Qwen3.6-27B ( nvfp4)
- MTP0 at a 7,680-token prompt: 11,191.5 prefill tok/s and86.4 decode tok/s. - MTP0 at a 260,096-token prompt: 2,510.6 prefill tok/s and59.9 decode tok/s. - MTP3 long reasoning: 213.1–231.0 decode tok/s with76.3–81.1% acceptance. - MTP3 structured output: 252.2 decode tok/s,** 89.8% acceptance**, and** 3.69 tokens/round**. - Against groupwise-int on the same corpus and runtime options: 3.48× the 7,680-token prefill throughput,** 1.55× the 260,096-token prefill throughput**, and** 30–32% higher MTP3 decode throughput**.
See Performance for the full methodology, variability, reproduction command, and per-fixture results.
Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):
| Model profile | AIME 2025 | AIME 2026 | GPQA-Diamond |
|---|---|---|---|
Qwen3.6-27B NVFP4Qwen3.6-35B-A3B groupwise-intQwen3.8-27B is supported but has not yet been added to this published evaluation campaign.
These are single-sample results under that NInfer evaluation profile, not pass@k. See the model cards and full performance document for correct/total counts and evaluation notes.
NInfer currently requires:
- 64-bit Linux;
- NVIDIA GeForce RTX 5090 (
sm_120a
); - NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;
- CMake 3.28 or newer and a C++20-capable host compiler;
pkg-config
;- FFmpeg development libraries:
libavformat >= 60
,libavcodec >= 60
,libavutil >= 58
, andlibswscale >= 7
; libcurl >= 7.85
;- Ninja, when using the commands below.
The build rejects CUDA architectures other than 120a
. There is no install target or packaged binary distribution; NInfer is run from its source build tree.
git clone https://github.com/Neroued/ninfer.git
cd ninfer
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
The default configuration builds:
build/apps/ninfer
build/apps/ninfer-serve
Tests, benchmarks, and maintainer tools are excluded from the default build.
Build the runtime image on a 64-bit Linux host with an RTX 5090, a CUDA 13.1-compatible NVIDIA driver, Docker, and the NVIDIA Container Toolkit.
docker build --tag ninfer:local .
Download a model into models/
as described below, then run the HTTP server:
docker run --rm \
--gpus '"device=0"' \
--publish 8080:8080 \
--volume "$PWD/models:/models:ro" \
ninfer:local \
ninfer-serve /models/qwen3_6_27b.ninfer \
--host 0.0.0.0
Run the CLI from the same image:
docker run --rm \
--gpus '"device=0"' \
--volume "$PWD/models:/models:ro" \
ninfer:local \
ninfer /models/qwen3_6_27b.ninfer \
--prompt "Explain prefill and decode in three sentences." \
--max-new 256
Use the Hugging Face CLI to download one of the registered artifacts:
hf download neroued/Qwen3.6-27B-NInfer \
qwen3_6_27b.ninfer \
--local-dir models
hf download neroued/Qwen3.6-27B-nvfp4-NInfer \
qwen3_6_27b_nvfp4.ninfer \
--local-dir models
hf download neroued/Qwen3.8-27B-NInfer \
qwen3_8_27b.ninfer \
--local-dir models
hf download neroued/Qwen3.6-35B-A3B-NInfer \
qwen3_6_35b_a3b.ninfer \
--local-dir models
Current NInfer builds accept only the version-2 artifact container, and all four downloads above are version 2. Migration applies only to Qwen3.6 artifacts downloaded before their version-2 publication; Qwen3.8-27B was published directly as version 2. Migrate an older exact local file in place:
python3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer
Use the same command with qwen3_6_27b_nvfp4.ninfer
or qwen3_6_35b_a3b.ninfer
for those artifacts. The migration updates only container metadata; it does not rewrite the weight payload. Alternatively, download the current version-2 file again from its Hugging Face repository.
Each .ninfer
file contains the weights and frontend resources needed by NInfer. It is not a Transformers checkpoint, Safetensors distribution, or GGUF file.
Each artifact is complete, while GPU residency is fixed at process startup. Speculative decoding is
disabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded.
Vision is also disabled by default, so its weights, Vision scratch phase, and frozen
request-transient allocation are omitted. Add --vision
to the CLI or server process that must accept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is available only for the 35B-A3B target and is text-only.
./build/apps/ninfer models/qwen3_6_27b.ninfer \
--prompt "Explain prefill and decode in three sentences." \
--max-context 16384 \
--max-new 256 \
--spec mtp --draft-tokens 3 \
--lm-head-draft
Use --messages FILE
instead of --prompt
for chat history, images, or videos:
./build/apps/ninfer models/qwen3_6_27b.ninfer \
--messages examples/cli/messages/image_chart.json \
--max-context 8192 \
--max-new 128 \
--vision
Answer content is written to stdout. progress, reasoning, timing, throughput, memory, and speculative-decoding statistics are written to stderr. See the CLI guide and committed examples for structured input and runtime options.
./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
--max-context 16384 \
--kv-capacity auto \
--max-concurrency 2 \
--spec mtp --draft-tokens 3 \
--lm-head-draft
The public model ID defaults to the artifact's identity.model_id
; use --model-id
only to publish a deployment-specific alias.
Then send an OpenAI-style request:
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3.6-27b",
"messages": [{"role": "user", "content": "Reply with one short sentence."}],
"max_tokens": 64
}'
The server also implements OpenAI Responses Core (typed Items, semantic SSE, local continuation state, and function calls) plus Anthropic Messages, token counting, and multimodal input. See HTTP serving.
All three registered model IDs support:
- text generation with thinking and non-thinking prompt modes;
- image, multi-image, video, and mixed multimodal messages;
- chunked prefill and CUDA Graph decode;
- startup-bounded small-scale concurrent serving with true batched decode;
- MTP speculative decoding with draft windows from one to five;
- BF16 and INT8 group-64 KV cache;
- model- and thinking-mode-aware official sampling defaults, with explicit greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty overrides;
- compatible-prefix reuse;
- OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming and usage accounting;
- prompt-rendered function tools and parsed tool calls.
The 35B-A3B target additionally supports text-only DFlash speculative decoding with draft windows from one to fifteen.
- Only the four
(model_id, weights_id)
artifact identities listed above are accepted product identities. - Execution is specialized for one RTX 5090 and one CUDA device.
- One Engine owns one resident model and supports a startup-fixed capacity of 1–8 active requests. Decode-ready requests are compacted at round boundaries and executed in one batched model traversal.
- NInfer does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.
--max-context
is the logical ceiling of each sequence and is configurable up to the registered models' native 262,144-token limit.--kv-capacity N
explicitly sizes the shared Main Text KV pool for all active and retained sequences, while--kv-capacity auto
selects the largest usable capacity from the memory remaining after weights are loaded while preserving 1 GiB of sizing headroom. Omission defaults to one--max-context
worth of pages. The resolved pool is fixed at startup and is not divided statically among request lanes.- Tool calls are parsed and returned to the client; NInfer does not execute tools.
- The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.
NInfer is licensed under the Apache License 2.0.
The published artifacts are derived from
Qwen/Qwen3.6-27B,
Qwen/Qwen3.8-27B, and
Qwen/Qwen3.6-35B-A3B. The 27B NVFP4 artifact also
uses the fixed packed weights from
rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm.
These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own
license files under third_party/
.