{"slug": "high-performance-single-gpu-inference-for-selected-model-checkpoints-and-gpus", "title": "High-performance single-GPU inference for selected model checkpoints and GPUs", "summary": "NInfer, a from-scratch C++/CUDA inference engine, achieves up to 1,313.8 aggregate decode tokens per second on a single NVIDIA GeForce RTX 5090 for the Qwen3.6-35B-A3B model at concurrency 8, with the Qwen3.6-27B NVFP4 profile reaching 1,146.9 tok/s. The engine supports explicitly registered Qwen checkpoints via local CLI or OpenAI-/Anthropic-compatible HTTP APIs.", "body_md": "Selected checkpoints. Maximum single-GPU inference performance.\n\nNInfer is a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs.\n\nNInfer deliberately supports a closed set of model artifacts instead of acting as a general model runtime:\n\n| Model | Weights | NInfer artifact | Size | SHA-256 |\n|---|---|---|---|---|\n|\n\n`groupwise-int`\n\n`qwen3_6_27b.ninfer`\n\n`7b51600ffd10632b9660f56085efdd9b751d79733ad32036a652234b64bebe7b`\n\n[Qwen3.6-27B NVFP4](https://huggingface.co/neroued/Qwen3.6-27B-nvfp4-NInfer)`nvfp4`\n\n`qwen3_6_27b_nvfp4.ninfer`\n\n`bce5f00d066c0f20f1317bf1fdcb458264cf95837c3b1f3fbec163694627893a`\n\n[Qwen3.8-27B](https://huggingface.co/neroued/Qwen3.8-27B-NInfer)`groupwise-int`\n\n`qwen3_8_27b.ninfer`\n\n`eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e`\n\n[Qwen3.6-35B-A3B](https://huggingface.co/neroued/Qwen3.6-35B-A3B-NInfer)`groupwise-int`\n\n`qwen3_6_35b_a3b.ninfer`\n\n`1fb9ea0b5b8561e49d9604115ec89e5d9f2b6f6434e32c37c57fffd480a325d2`\n\nThe two Qwen3.6-27B weight profiles bind to the registered `qwen3_6_27b`\n\ntarget; the version-2\nartifact identity selects the profile without a separate runtime flag. Qwen3.8-27B is separately\nregistered as `qwen3_8_27b`\n\nand shares the 27B execution package while using W8 token-embedding and\nfull-output-head weights. The `nvfp4`\n\nprofile uses W4A4 Tensor Core MMA for prefill and A16 NVFP4\nkernels for decode. All three 27B artifacts retain the same Text, Vision, MTP, prefix-reuse, CLI,\nand serving routes.\n\nThe published measurements currently cover the three Qwen3.6 artifact profiles. Qwen3.8-27B is supported by current NInfer builds but is not yet included in the benchmark campaign.\n\nSaturated decode was measured on an RTX 5090 with INT8 group-64 KV cache, CUDA Graphs, MTP3, and one 8,192-token generation per active request. The values below are aggregate committed decode throughput from complete one-second intervals in which the actual decode batch remained equal to the configured concurrency. Each profile should be read independently.\n\n| Model profile | C=1 | C=2 | C=4 | C=8 | C8 / C1 |\n|---|---|---|---|---|---|\nQwen3.6-27B `groupwise-int` |\n185.8 tok/s | 247.0 tok/s | 309.5 tok/s | 535.0 tok/s | 2.88× |\nQwen3.6-27B `nvfp4` |\n202.4 tok/s | 399.7 tok/s | 699.7 tok/s | 1,146.9 tok/s | 5.67× |\nQwen3.6-35B-A3B `groupwise-int` |\n593.0 tok/s | 877.7 tok/s | 1,166.0 tok/s | 1,313.8 tok/s | 2.22× |\n\nAt C=8, Qwen3.6-35B-A3B reaches **1,313.8 aggregate decode tok/s**. The 27B NVFP4 profile reaches\n**1,146.9 tok/s** and **5.67×** its C=1 throughput.\n\nThe single-request corpus was measured on the same GPU with INT8 group-64 KV cache, CUDA Graphs, and a 1,024-token prefill chunk. Each reported fixture uses five fixed seeds after server warm-up. The two measured targets are reported independently and are not cross-target comparisons. The two 27B weight profiles are reported separately. Requests were submitted serially to a persistent server.\n\n**Qwen3.6-35B-A3B**\n\n- MTP0 at a 7,680-token prompt:\n**15,544.3 prefill tok/s** and**271.1 decode tok/s**. - MTP0 at a 260,096-token prompt:\n**5,157.1 prefill tok/s** and**188.2 decode tok/s**. - MTP3 long reasoning:\n**620.3–726.2 decode tok/s** with**72.7–82.8% acceptance**. - MTP3 structured output:\n**770.9 decode tok/s**,** 89.1% acceptance**, and** 3.67 tokens/round**.\n\n**Qwen3.6-27B ( groupwise-int)**\n\n- MTP0 at a 7,680-token prompt:\n**3,218.1 prefill tok/s** and**77.6 decode tok/s**. - MTP0 at a 260,096-token prompt:\n**1,614.8 prefill tok/s** and**54.8 decode tok/s**. - MTP3 long reasoning:\n**161.9–175.4 decode tok/s** with**73.4–78.8% acceptance**. - MTP3 structured output:\n**193.0 decode tok/s**,** 88.7% acceptance**, and** 3.66 tokens/round**.\n\n**Qwen3.6-27B ( nvfp4)**\n\n- MTP0 at a 7,680-token prompt:\n**11,191.5 prefill tok/s** and**86.4 decode tok/s**. - MTP0 at a 260,096-token prompt:\n**2,510.6 prefill tok/s** and**59.9 decode tok/s**. - MTP3 long reasoning:\n**213.1–231.0 decode tok/s** with**76.3–81.1% acceptance**. - MTP3 structured output:\n**252.2 decode tok/s**,** 89.8% acceptance**, and** 3.69 tokens/round**. - Against groupwise-int on the same corpus and runtime options:\n**3.48× the 7,680-token prefill throughput**,** 1.55× the 260,096-token prefill throughput**, and** 30–32% higher MTP3 decode throughput**.\n\nSee [Performance](/Neroued/ninfer/blob/master/docs/performance.md) for the full methodology, variability, reproduction command,\nand per-fixture results.\n\nCapability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):\n\n| Model profile | AIME 2025 | AIME 2026 | GPQA-Diamond |\n|---|---|---|---|\n|\n\n[Qwen3.6-27B NVFP4](/Neroued/ninfer/blob/master/model-cards/Qwen3.6-27B-nvfp4-NInfer/README.md)[Qwen3.6-35B-A3B groupwise-int](/Neroued/ninfer/blob/master/model-cards/Qwen3.6-35B-A3B-NInfer/README.md)Qwen3.8-27B is supported but has not yet been added to this published evaluation campaign.\n\nThese are single-sample results under that NInfer evaluation profile, not pass@k. See the model\ncards and [full performance document](/Neroued/ninfer/blob/master/docs/performance.md) for correct/total counts and evaluation\nnotes.\n\nNInfer currently requires:\n\n- 64-bit Linux;\n- NVIDIA GeForce RTX 5090 (\n`sm_120a`\n\n); - NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;\n- CMake 3.28 or newer and a C++20-capable host compiler;\n`pkg-config`\n\n;- FFmpeg development libraries:\n`libavformat >= 60`\n\n,`libavcodec >= 60`\n\n,`libavutil >= 58`\n\n, and`libswscale >= 7`\n\n; `libcurl >= 7.85`\n\n;- Ninja, when using the commands below.\n\nThe build rejects CUDA architectures other than `120a`\n\n. There is no install target or packaged\nbinary distribution; NInfer is run from its source build tree.\n\n```\ngit clone https://github.com/Neroued/ninfer.git\ncd ninfer\n\ncmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release\ncmake --build build --parallel\n```\n\nThe default configuration builds:\n\n```\nbuild/apps/ninfer\nbuild/apps/ninfer-serve\n```\n\nTests, benchmarks, and maintainer tools are excluded from the default build.\n\nBuild the runtime image on a 64-bit Linux host with an RTX 5090, a CUDA 13.1-compatible NVIDIA\ndriver, Docker, and the\n[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).\n\n```\ndocker build --tag ninfer:local .\n```\n\nDownload a model into `models/`\n\nas described below, then run the HTTP server:\n\n```\ndocker run --rm \\\n  --gpus '\"device=0\"' \\\n  --publish 8080:8080 \\\n  --volume \"$PWD/models:/models:ro\" \\\n  ninfer:local \\\n  ninfer-serve /models/qwen3_6_27b.ninfer \\\n  --host 0.0.0.0\n```\n\nRun the CLI from the same image:\n\n```\ndocker run --rm \\\n  --gpus '\"device=0\"' \\\n  --volume \"$PWD/models:/models:ro\" \\\n  ninfer:local \\\n  ninfer /models/qwen3_6_27b.ninfer \\\n  --prompt \"Explain prefill and decode in three sentences.\" \\\n  --max-new 256\n```\n\nUse the Hugging Face CLI to download one of the registered artifacts:\n\n```\nhf download neroued/Qwen3.6-27B-NInfer \\\n  qwen3_6_27b.ninfer \\\n  --local-dir models\n\n# Or the 27B NVFP4 weight variant:\nhf download neroued/Qwen3.6-27B-nvfp4-NInfer \\\n  qwen3_6_27b_nvfp4.ninfer \\\n  --local-dir models\n\n# Or Qwen3.8-27B:\nhf download neroued/Qwen3.8-27B-NInfer \\\n  qwen3_8_27b.ninfer \\\n  --local-dir models\n\n# Or:\nhf download neroued/Qwen3.6-35B-A3B-NInfer \\\n  qwen3_6_35b_a3b.ninfer \\\n  --local-dir models\n```\n\nCurrent NInfer builds accept only the version-2 artifact container, and all four downloads above are version 2. Migration applies only to Qwen3.6 artifacts downloaded before their version-2 publication; Qwen3.8-27B was published directly as version 2. Migrate an older exact local file in place:\n\n```\npython3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer\n```\n\nUse the same command with `qwen3_6_27b_nvfp4.ninfer`\n\nor `qwen3_6_35b_a3b.ninfer`\n\nfor those\nartifacts. The migration updates only container metadata; it does not rewrite the weight payload.\nAlternatively, download the current version-2 file again from its Hugging Face repository.\n\nEach `.ninfer`\n\nfile contains the weights and frontend resources needed by NInfer. It is not a\nTransformers checkpoint, Safetensors distribution, or GGUF file.\n\nEach artifact is complete, while GPU residency is fixed at process startup. Speculative decoding is\ndisabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded.\nVision is also disabled by default, so its weights, Vision scratch phase, and frozen\nrequest-transient allocation are omitted. Add `--vision`\n\nto the CLI or server process that must\naccept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is\navailable only for the 35B-A3B target and is text-only.\n\n```\n./build/apps/ninfer models/qwen3_6_27b.ninfer \\\n  --prompt \"Explain prefill and decode in three sentences.\" \\\n  --max-context 16384 \\\n  --max-new 256 \\\n  --spec mtp --draft-tokens 3 \\\n  --lm-head-draft\n```\n\nUse `--messages FILE`\n\ninstead of `--prompt`\n\nfor chat history, images, or videos:\n\n```\n./build/apps/ninfer models/qwen3_6_27b.ninfer \\\n  --messages examples/cli/messages/image_chart.json \\\n  --max-context 8192 \\\n  --max-new 128 \\\n  --vision\n```\n\nAnswer content is written to stdout. Loading progress, reasoning, timing, throughput, memory, and\nspeculative-decoding statistics are written to stderr. See the [CLI guide](/Neroued/ninfer/blob/master/docs/cli.md) and\n[committed examples](/Neroued/ninfer/blob/master/examples/cli) for structured input and runtime options.\n\n```\n./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \\\n  --max-context 16384 \\\n  --kv-capacity auto \\\n  --max-concurrency 2 \\\n  --spec mtp --draft-tokens 3 \\\n  --lm-head-draft\n```\n\nThe public model ID defaults to the artifact's `identity.model_id`\n\n; use `--model-id`\n\nonly to\npublish a deployment-specific alias.\n\nThen send an OpenAI-style request:\n\n```\ncurl http://127.0.0.1:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"qwen3.6-27b\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Reply with one short sentence.\"}],\n    \"max_tokens\": 64\n  }'\n```\n\nThe server also implements OpenAI Responses Core (typed Items, semantic SSE, local continuation\nstate, and function calls) plus Anthropic Messages, token counting, and multimodal input. See\n[HTTP serving](/Neroued/ninfer/blob/master/docs/serving.md).\n\nAll three registered model IDs support:\n\n- text generation with thinking and non-thinking prompt modes;\n- image, multi-image, video, and mixed multimodal messages;\n- chunked prefill and CUDA Graph decode;\n- startup-bounded small-scale concurrent serving with true batched decode;\n- MTP speculative decoding with draft windows from one to five;\n- BF16 and INT8 group-64 KV cache;\n- model- and thinking-mode-aware official sampling defaults, with explicit greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty overrides;\n- compatible-prefix reuse;\n- OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming and usage accounting;\n- prompt-rendered function tools and parsed tool calls.\n\nThe 35B-A3B target additionally supports text-only DFlash speculative decoding with draft windows from one to fifteen.\n\n- Only the four\n`(model_id, weights_id)`\n\nartifact identities listed above are accepted product identities. - Execution is specialized for one RTX 5090 and one CUDA device.\n- One Engine owns one resident model and supports a startup-fixed capacity of 1–8 active requests. Decode-ready requests are compacted at round boundaries and executed in one batched model traversal.\n- NInfer does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.\n`--max-context`\n\nis the logical ceiling of each sequence and is configurable up to the registered models' native 262,144-token limit.`--kv-capacity N`\n\nexplicitly sizes the shared Main Text KV pool for all active and retained sequences, while`--kv-capacity auto`\n\nselects the largest usable capacity from the memory remaining after weights are loaded while preserving 1 GiB of sizing headroom. Omission defaults to one`--max-context`\n\nworth of pages. The resolved pool is fixed at startup and is not divided statically among request lanes.- Tool calls are parsed and returned to the client; NInfer does not execute tools.\n- The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.\n\nNInfer is licensed under the [Apache License 2.0](/Neroued/ninfer/blob/master/LICENSE).\n\nThe published artifacts are derived from\n[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B),\n[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), and\n[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). The 27B NVFP4 artifact also\nuses the fixed packed weights from\n[rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm](https://huggingface.co/rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm).\nThese source repositories are distributed under Apache-2.0. Vendored dependencies retain their own\nlicense files under `third_party/`\n\n.", "url": "https://wpnews.pro/news/high-performance-single-gpu-inference-for-selected-model-checkpoints-and-gpus", "canonical_source": "https://github.com/Neroued/ninfer", "published_at": "2026-08-15 07:45:44+00:00", "updated_at": "2026-08-15 08:11:26.305422+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["NInfer", "NVIDIA GeForce RTX 5090", "Qwen3.6-35B-A3B", "Qwen3.6-27B", "Qwen3.8-27B"], "alternates": {"html": "https://wpnews.pro/news/high-performance-single-gpu-inference-for-selected-model-checkpoints-and-gpus", "markdown": "https://wpnews.pro/news/high-performance-single-gpu-inference-for-selected-model-checkpoints-and-gpus.md", "text": "https://wpnews.pro/news/high-performance-single-gpu-inference-for-selected-model-checkpoints-and-gpus.txt", "jsonld": "https://wpnews.pro/news/high-performance-single-gpu-inference-for-selected-model-checkpoints-and-gpus.jsonld"}}