Performant C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 / Blackwell

A developer released a hyper-optimized C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 Blackwell GPUs, achieving 13.4k tokens/sec prefill and 270+ tokens/sec decode, outperforming generic runtimes like llama.cpp. The engine features hybrid architecture support, dual-tier state management, and an OpenAI-compatible server.

A hyper-optimized, zero-dependency C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 / Blackwell. A zero-dependency C/CUDA codebase specialized for a single model + GPU pairing — Qwen3.6-35B-A3B MXFP4 GGUF on RTX 5090 sm 120a — on the bet that a dedicated engine beats generic runtimes llama.cpp / vLLM / SGLang on their long tail. Status: faster than llama.cpp at every measured point. - 🚀 Extreme Prefill Throughput : Tensor-core FlashAttention prefill reaching 13.4k tokens/sec at context depth 2,048, measured on a GPU clocked at 400W and 7.6k tokens/sec at context depth 90k . - ⚡ Fast Decode : Native token generation speeds of 270+ tokens/sec using captured CUDA graphs with zero host syncs. - 🧠 Hybrid Architecture Support : Native CUDA implementations for both 10 full-attention layers using W4A8 block-scaled MMA MoE and 30 recurrent SSM layers gated-DeltaNet scans . - 💾 Dual-Tier State Management : Zero-overhead VRAM saving 2.5× smaller KV cache via --kv-quant and DRAM/Disk state checkpoint caching restoring context states in 3ms . - 🌐 OpenAI-Compatible Server : A zero-dependency, prefix-cached HTTP server q36 server supporting SSE streaming and tool calling. - 🛠️ Developer Tooling : Built-in benchmark tools q36 bench , perplexity evaluation harnesses q36 ppl , and CPU-only validation helpers. GPU : An NVIDIA RTX 5090 or RTX 6000 Pro Blackwell GPU Blackwell architecture, sm 120a / compute capability 12.0+ is required . Note: The engine is architecturally compatible with the RTX 6000 Pro Blackwell and benefits from its 96 GB VRAM , but has not yet been physically tested on it. Software : Linux with CUDA Toolkit 13.1 https://developer.nvidia.com/cuda-downloads installed — the version the engine is developed and tested on 12.x releases that expose sm 120a may work but are untested . Ensure the CUDA compiler nvcc is on your system PATH . Model : Download the model weights in MXFP4 GGUF format from Hugging Face: unsloth/Qwen3.6-35B-A3B-MTP-GGUF https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF . Run the following commands from the repository root: Compile and run validation tests on CPU no GPU required : make tools ./q36 info /path/to/model.gguf Compile the full GPU-accelerated engine, benchmark, and OpenAI server: make q36 q36 bench q36 server Interactive CLI : Run the interactive chat shell: ./q36 -m /path/to/model.gguf OpenAI Server : Start the HTTP API server with state caching enabled: ./q36 server -m /path/to/model.gguf --port 8080 --ctx 32768 Benchmark : Run throughput tests: ./q36 bench -m /path/to/model.gguf Due to the hybrid model architecture 10 attention layers + 30 recurrent SSM layers , q36 optimizes KV cache and state memory at two levels: By default, attention keys and values are stored in FP16. Toggle quantization with the --kv-quant flag: Mechanism : Quantizes keys to Q8 0 8-bit and values to MXFP4 4-bit block-scaled FP4 . VRAM savings : Reduces active VRAM KV cache size by 2.5× , freeing up space for long contexts and larger batches. Performance trade-offs : Short contexts : Adds small kernel overhead ~5% slower decoding . Long contexts 30k+ tokens : Speeds up generation. Since the quantized cache transfers 40% of the bytes of FP16, it overcomes memory-bandwidth bottlenecks. Accuracy cost : Negligible perplexity increase +0.54% . Standard prefix caching fails in multi-turn agent/tool loops when prompts are reconstructed and resent. q36 server provides a State Checkpoint Cache to offload recurrent states and KV pages: Mechanism : At each prompt end, the server snapshots the live engine state attention KV + the ~63MB recurrent SSM state to system DRAM. If the next prompt shares a prefix, the state is restored from DRAM in 3ms instead of triggering a multi-second re-prefill. Configuration Flags : --cache-ram MB : Memory allocated for DRAM cache defaults to auto: ~4 full-context checkpoints, capped at 25% of system RAM, floor 4GB . --cache-min tokens : Minimum prefix token length to trigger checkpointing default: 2048 . --cache-dir path : Local directory path to write evicted checkpoints to disk LRU tiering . Checkpoints persisted here survive server restarts. --no-state-cache : Disables checkpoint caching completely. The underlying q36 engine supports high-throughput, multi-tenant execution using continuous batching and slot management primitives. While the OpenAI HTTP server q36 server currently processes requests sequentially on Slot 0 , the core library exposes a full multi-tenant scheduler API: q36 engine create mt model, max ctx, n slots : Instantiates an engine with n slots concurrent sequence states. Attention KV cache pages and recurrent SSM state blocks are separate per slot. q36 engine prefill slot engine, slot, tokens, len, pos0 : Processes prompts into a specific slot. q36 engine step active engine, n active, active tokens, positions, out tokens : Performs a single batched decode step for all n active slots. To avoid graph launch overheads, active slots are padded/bucketed to {8, 16, 32, 48, 64} to reuse captured CUDA graphs. q36 engine slot move engine, dst slot, src slot : Instantly relocates a slot's entire history attention KV, SSM state, convolution buffers in memory ~40us overhead . This allows slot compaction upon request eviction to maintain a contiguous active slot block. The engine dynamically switches decoding strategies based on the active batch size: GEMV Mode Batch < 16 : Lowest latency; serial state updates. Batch-Tiled Mode Batch = 16 : Uses tensor cores and tiled GEMM kernels to load and share weight matrices across all active sequences, achieving aggregate throughput of up to 1,653 tokens/sec at Batch=64. The engine automatically transitions between these modes, which can also be forced via Q36 MT GEMV / Q36 MT TILED . You can simulate staggered request arrival, execution, and eviction utilizing slot compaction under churn via the benchmark tool: Run a continuous batching simulation with up to 48 concurrent slots: Q36 CB=48 Q36 CB NREQ=256 ./q36 bench -m /path/to/model.gguf q36 engine.cu /ambud/q36/blob/main/q36 engine.cu : Core engine loop, CUDA graph setup, Blackwell W4A8 MMA kernels, and FlashAttention. q36 dequant.cuh /ambud/q36/blob/main/q36 dequant.cuh : Quantization formats, layouts, and CPU-parallelized dequantization. tokenizer.c /ambud/q36/blob/main/tokenizer.c : Native BPE tokenizer implementation. server.c /ambud/q36/blob/main/server.c : Zero-dependency OpenAI-compatible HTTP server. ARCHITECTURE.md /ambud/q36/blob/main/docs/ARCHITECTURE.md : Systems deep dive — decode roofline analysis, Blackwell block-scaled MMA, the SSM/attention caching asymmetry, multi-GPU trade-offs. ENGINE.md /ambud/q36/blob/main/docs/ENGINE.md : Detailed engine reference — benchmarks, CLI/server flags, project layout. This project is licensed under the AGPL-3.0 License /ambud/q36/blob/main/LICENSE . If you run a modified version of these engines as a network service, you must make your modified source available to its users. Copyright is retained by Ambud Sharma https://github.com/ambud . The AGPL applies to everyone else's use of this code; Ambud Sharma reserves the right to offer this software under other license terms dual licensing . Contributions require the Contributor License Agreement /ambud/q36/blob/main/CLA.md see CONTRIBUTING.md /ambud/q36/blob/main/CONTRIBUTING.md , preserving the project's ability to dual-license. This codebase contains third-party components adapted from llama.cpp licensed under the MIT License : - Dequantization constants, block layout structures, and scaling logic in q36 dequant.cuh /ambud/q36/blob/main/q36 dequant.cuh . - Perplexity calculation windowing strategies in ppl.c /ambud/q36/blob/main/ppl.c . The full MIT License text and copyright notices for these components are preserved in the respective source files. Built by Ambud Sharma https://github.com/ambud — connect with me on LinkedIn https://www.linkedin.com/in/ambud/ .