# High-performance single-GPU inference for selected model checkpoints and GPUs

> Source: <https://github.com/Neroued/ninfer>
> Published: 2026-08-15 07:45:44+00:00

Selected checkpoints. Maximum single-GPU inference performance.

NInfer is a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs.

NInfer deliberately supports a closed set of model artifacts instead of acting as a general model runtime:

| Model | Weights | NInfer artifact | Size | SHA-256 |
|---|---|---|---|---|
|

`groupwise-int`

`qwen3_6_27b.ninfer`

`7b51600ffd10632b9660f56085efdd9b751d79733ad32036a652234b64bebe7b`

[Qwen3.6-27B NVFP4](https://huggingface.co/neroued/Qwen3.6-27B-nvfp4-NInfer)`nvfp4`

`qwen3_6_27b_nvfp4.ninfer`

`bce5f00d066c0f20f1317bf1fdcb458264cf95837c3b1f3fbec163694627893a`

[Qwen3.8-27B](https://huggingface.co/neroued/Qwen3.8-27B-NInfer)`groupwise-int`

`qwen3_8_27b.ninfer`

`eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e`

[Qwen3.6-35B-A3B](https://huggingface.co/neroued/Qwen3.6-35B-A3B-NInfer)`groupwise-int`

`qwen3_6_35b_a3b.ninfer`

`1fb9ea0b5b8561e49d9604115ec89e5d9f2b6f6434e32c37c57fffd480a325d2`

The two Qwen3.6-27B weight profiles bind to the registered `qwen3_6_27b`

target; the version-2
artifact identity selects the profile without a separate runtime flag. Qwen3.8-27B is separately
registered as `qwen3_8_27b`

and shares the 27B execution package while using W8 token-embedding and
full-output-head weights. The `nvfp4`

profile uses W4A4 Tensor Core MMA for prefill and A16 NVFP4
kernels for decode. All three 27B artifacts retain the same Text, Vision, MTP, prefix-reuse, CLI,
and serving routes.

The published measurements currently cover the three Qwen3.6 artifact profiles. Qwen3.8-27B is supported by current NInfer builds but is not yet included in the benchmark campaign.

Saturated decode was measured on an RTX 5090 with INT8 group-64 KV cache, CUDA Graphs, MTP3, and one 8,192-token generation per active request. The values below are aggregate committed decode throughput from complete one-second intervals in which the actual decode batch remained equal to the configured concurrency. Each profile should be read independently.

| Model profile | C=1 | C=2 | C=4 | C=8 | C8 / C1 |
|---|---|---|---|---|---|
Qwen3.6-27B `groupwise-int` |
185.8 tok/s | 247.0 tok/s | 309.5 tok/s | 535.0 tok/s | 2.88× |
Qwen3.6-27B `nvfp4` |
202.4 tok/s | 399.7 tok/s | 699.7 tok/s | 1,146.9 tok/s | 5.67× |
Qwen3.6-35B-A3B `groupwise-int` |
593.0 tok/s | 877.7 tok/s | 1,166.0 tok/s | 1,313.8 tok/s | 2.22× |

At C=8, Qwen3.6-35B-A3B reaches **1,313.8 aggregate decode tok/s**. The 27B NVFP4 profile reaches
**1,146.9 tok/s** and **5.67×** its C=1 throughput.

The single-request corpus was measured on the same GPU with INT8 group-64 KV cache, CUDA Graphs, and a 1,024-token prefill chunk. Each reported fixture uses five fixed seeds after server warm-up. The two measured targets are reported independently and are not cross-target comparisons. The two 27B weight profiles are reported separately. Requests were submitted serially to a persistent server.

**Qwen3.6-35B-A3B**

- MTP0 at a 7,680-token prompt:
**15,544.3 prefill tok/s** and**271.1 decode tok/s**. - MTP0 at a 260,096-token prompt:
**5,157.1 prefill tok/s** and**188.2 decode tok/s**. - MTP3 long reasoning:
**620.3–726.2 decode tok/s** with**72.7–82.8% acceptance**. - MTP3 structured output:
**770.9 decode tok/s**,** 89.1% acceptance**, and** 3.67 tokens/round**.

**Qwen3.6-27B ( groupwise-int)**

- MTP0 at a 7,680-token prompt:
**3,218.1 prefill tok/s** and**77.6 decode tok/s**. - MTP0 at a 260,096-token prompt:
**1,614.8 prefill tok/s** and**54.8 decode tok/s**. - MTP3 long reasoning:
**161.9–175.4 decode tok/s** with**73.4–78.8% acceptance**. - MTP3 structured output:
**193.0 decode tok/s**,** 88.7% acceptance**, and** 3.66 tokens/round**.

**Qwen3.6-27B ( nvfp4)**

- MTP0 at a 7,680-token prompt:
**11,191.5 prefill tok/s** and**86.4 decode tok/s**. - MTP0 at a 260,096-token prompt:
**2,510.6 prefill tok/s** and**59.9 decode tok/s**. - MTP3 long reasoning:
**213.1–231.0 decode tok/s** with**76.3–81.1% acceptance**. - MTP3 structured output:
**252.2 decode tok/s**,** 89.8% acceptance**, and** 3.69 tokens/round**. - Against groupwise-int on the same corpus and runtime options:
**3.48× the 7,680-token prefill throughput**,** 1.55× the 260,096-token prefill throughput**, and** 30–32% higher MTP3 decode throughput**.

See [Performance](/Neroued/ninfer/blob/master/docs/performance.md) for the full methodology, variability, reproduction command,
and per-fixture results.

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

| Model profile | AIME 2025 | AIME 2026 | GPQA-Diamond |
|---|---|---|---|
|

[Qwen3.6-27B NVFP4](/Neroued/ninfer/blob/master/model-cards/Qwen3.6-27B-nvfp4-NInfer/README.md)[Qwen3.6-35B-A3B groupwise-int](/Neroued/ninfer/blob/master/model-cards/Qwen3.6-35B-A3B-NInfer/README.md)Qwen3.8-27B is supported but has not yet been added to this published evaluation campaign.

These are single-sample results under that NInfer evaluation profile, not pass@k. See the model
cards and [full performance document](/Neroued/ninfer/blob/master/docs/performance.md) for correct/total counts and evaluation
notes.

NInfer currently requires:

- 64-bit Linux;
- NVIDIA GeForce RTX 5090 (
`sm_120a`

); - NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;
- CMake 3.28 or newer and a C++20-capable host compiler;
`pkg-config`

;- FFmpeg development libraries:
`libavformat >= 60`

,`libavcodec >= 60`

,`libavutil >= 58`

, and`libswscale >= 7`

; `libcurl >= 7.85`

;- Ninja, when using the commands below.

The build rejects CUDA architectures other than `120a`

. There is no install target or packaged
binary distribution; NInfer is run from its source build tree.

```
git clone https://github.com/Neroued/ninfer.git
cd ninfer

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
```

The default configuration builds:

```
build/apps/ninfer
build/apps/ninfer-serve
```

Tests, benchmarks, and maintainer tools are excluded from the default build.

Build the runtime image on a 64-bit Linux host with an RTX 5090, a CUDA 13.1-compatible NVIDIA
driver, Docker, and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).

```
docker build --tag ninfer:local .
```

Download a model into `models/`

as described below, then run the HTTP server:

```
docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_6_27b.ninfer \
  --host 0.0.0.0
```

Run the CLI from the same image:

```
docker run --rm \
  --gpus '"device=0"' \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer /models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-new 256
```

Use the Hugging Face CLI to download one of the registered artifacts:

```
hf download neroued/Qwen3.6-27B-NInfer \
  qwen3_6_27b.ninfer \
  --local-dir models

# Or the 27B NVFP4 weight variant:
hf download neroued/Qwen3.6-27B-nvfp4-NInfer \
  qwen3_6_27b_nvfp4.ninfer \
  --local-dir models

# Or Qwen3.8-27B:
hf download neroued/Qwen3.8-27B-NInfer \
  qwen3_8_27b.ninfer \
  --local-dir models

# Or:
hf download neroued/Qwen3.6-35B-A3B-NInfer \
  qwen3_6_35b_a3b.ninfer \
  --local-dir models
```

Current NInfer builds accept only the version-2 artifact container, and all four downloads above are version 2. Migration applies only to Qwen3.6 artifacts downloaded before their version-2 publication; Qwen3.8-27B was published directly as version 2. Migrate an older exact local file in place:

```
python3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer
```

Use the same command with `qwen3_6_27b_nvfp4.ninfer`

or `qwen3_6_35b_a3b.ninfer`

for those
artifacts. The migration updates only container metadata; it does not rewrite the weight payload.
Alternatively, download the current version-2 file again from its Hugging Face repository.

Each `.ninfer`

file contains the weights and frontend resources needed by NInfer. It is not a
Transformers checkpoint, Safetensors distribution, or GGUF file.

Each artifact is complete, while GPU residency is fixed at process startup. Speculative decoding is
disabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded.
Vision is also disabled by default, so its weights, Vision scratch phase, and frozen
request-transient allocation are omitted. Add `--vision`

to the CLI or server process that must
accept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is
available only for the 35B-A3B target and is text-only.

```
./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 16384 \
  --max-new 256 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft
```

Use `--messages FILE`

instead of `--prompt`

for chat history, images, or videos:

```
./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --messages examples/cli/messages/image_chart.json \
  --max-context 8192 \
  --max-new 128 \
  --vision
```

Answer content is written to stdout. Loading progress, reasoning, timing, throughput, memory, and
speculative-decoding statistics are written to stderr. See the [CLI guide](/Neroued/ninfer/blob/master/docs/cli.md) and
[committed examples](/Neroued/ninfer/blob/master/examples/cli) for structured input and runtime options.

```
./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
  --max-context 16384 \
  --kv-capacity auto \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft
```

The public model ID defaults to the artifact's `identity.model_id`

; use `--model-id`

only to
publish a deployment-specific alias.

Then send an OpenAI-style request:

```
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'
```

The server also implements OpenAI Responses Core (typed Items, semantic SSE, local continuation
state, and function calls) plus Anthropic Messages, token counting, and multimodal input. See
[HTTP serving](/Neroued/ninfer/blob/master/docs/serving.md).

All three registered model IDs support:

- text generation with thinking and non-thinking prompt modes;
- image, multi-image, video, and mixed multimodal messages;
- chunked prefill and CUDA Graph decode;
- startup-bounded small-scale concurrent serving with true batched decode;
- MTP speculative decoding with draft windows from one to five;
- BF16 and INT8 group-64 KV cache;
- model- and thinking-mode-aware official sampling defaults, with explicit greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty overrides;
- compatible-prefix reuse;
- OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming and usage accounting;
- prompt-rendered function tools and parsed tool calls.

The 35B-A3B target additionally supports text-only DFlash speculative decoding with draft windows from one to fifteen.

- Only the four
`(model_id, weights_id)`

artifact identities listed above are accepted product identities. - Execution is specialized for one RTX 5090 and one CUDA device.
- One Engine owns one resident model and supports a startup-fixed capacity of 1–8 active requests. Decode-ready requests are compacted at round boundaries and executed in one batched model traversal.
- NInfer does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.
`--max-context`

is the logical ceiling of each sequence and is configurable up to the registered models' native 262,144-token limit.`--kv-capacity N`

explicitly sizes the shared Main Text KV pool for all active and retained sequences, while`--kv-capacity auto`

selects the largest usable capacity from the memory remaining after weights are loaded while preserving 1 GiB of sizing headroom. Omission defaults to one`--max-context`

worth of pages. The resolved pool is fixed at startup and is not divided statically among request lanes.- Tool calls are parsed and returned to the client; NInfer does not execute tools.
- The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.

NInfer is licensed under the [Apache License 2.0](/Neroued/ninfer/blob/master/LICENSE).

The published artifacts are derived from
[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B),
[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), and
[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). The 27B NVFP4 artifact also
uses the fixed packed weights from
[rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm](https://huggingface.co/rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm).
These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own
license files under `third_party/`

.