{"slug": "performant-c-cuda-inference-engine-for-qwen-3-6-35b-on-rtx-5090-blackwell", "title": "Performant C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 / Blackwell", "summary": "A developer released a hyper-optimized C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 Blackwell GPUs, achieving 13.4k tokens/sec prefill and 270+ tokens/sec decode, outperforming generic runtimes like llama.cpp. The engine features hybrid architecture support, dual-tier state management, and an OpenAI-compatible server.", "body_md": "A hyper-optimized, zero-dependency C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 / Blackwell.\n\nA zero-dependency C/CUDA codebase specialized for a single model + GPU pairing — Qwen3.6-35B-A3B (MXFP4 GGUF) on RTX 5090 (sm_120a) — on the bet that a dedicated engine beats generic runtimes (llama.cpp / vLLM / SGLang) on their long tail. **Status: faster than llama.cpp at every measured point.**\n\n- 🚀\n**Extreme Prefill Throughput**: Tensor-core FlashAttention prefill reaching** 13.4k tokens/sec**(at context depth 2,048, measured on a GPU clocked at 400W) and** 7.6k tokens/sec**(at context depth 90k). - ⚡\n**Fast Decode**: Native token generation speeds of** 270+ tokens/sec**using captured CUDA graphs with zero host syncs. - 🧠\n**Hybrid Architecture Support**: Native CUDA implementations for both 10 full-attention layers (using W4A8 block-scaled MMA MoE) and 30 recurrent SSM layers (gated-DeltaNet scans). - 💾\n**Dual-Tier State Management**: Zero-overhead VRAM saving (** 2.5× smaller KV cache**via`--kv-quant`\n\n) and DRAM/Disk state checkpoint caching (restoring context states in**3ms**). - 🌐\n**OpenAI-Compatible Server**: A zero-dependency, prefix-cached HTTP server (`q36_server`\n\n) supporting SSE streaming and tool calling. - 🛠️\n**Developer Tooling**: Built-in benchmark tools (`q36_bench`\n\n), perplexity evaluation harnesses (`q36_ppl`\n\n), and CPU-only validation helpers.\n\n**GPU**: An NVIDIA RTX 5090 or RTX 6000 Pro Blackwell GPU (Blackwell architecture,`sm_120a`\n\n/ compute capability 12.0+ is required).*Note: The engine is architecturally compatible with the RTX 6000 Pro Blackwell (and benefits from its 96 GB VRAM), but has not yet been physically tested on it.***Software**: Linux with[CUDA Toolkit 13.1](https://developer.nvidia.com/cuda-downloads)installed — the version the engine is developed and tested on (12.x releases that expose`sm_120a`\n\nmay work but are untested). Ensure the CUDA compiler (`nvcc`\n\n) is on your system`PATH`\n\n.**Model**: Download the model weights in MXFP4 GGUF format from[Hugging Face: unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF).\n\nRun the following commands from the repository root:\n\n```\n# Compile and run validation tests on CPU (no GPU required):\nmake tools\n./q36_info /path/to/model.gguf\n\n# Compile the full GPU-accelerated engine, benchmark, and OpenAI server:\nmake q36 q36_bench q36_server\n```\n\n**Interactive CLI**: Run the interactive chat shell:\n\n```\n./q36 -m /path/to/model.gguf\n```\n\n**OpenAI Server**: Start the HTTP API server with state caching enabled:\n\n```\n./q36_server -m /path/to/model.gguf --port 8080 --ctx 32768\n```\n\n**Benchmark**: Run throughput tests:\n\n```\n./q36_bench -m /path/to/model.gguf\n```\n\nDue to the hybrid model architecture (10 attention layers + 30 recurrent SSM layers), `q36`\n\noptimizes KV cache and state memory at two levels:\n\nBy default, attention keys and values are stored in FP16. Toggle quantization with the `--kv-quant`\n\nflag:\n\n**Mechanism**: Quantizes keys to** Q8_0**(8-bit) and values to** MXFP4**(4-bit block-scaled FP4).** VRAM savings**: Reduces active VRAM KV cache size by** 2.5×**, freeing up space for long contexts and larger batches.** Performance trade-offs**:*Short contexts*: Adds small kernel overhead (~5% slower decoding).*Long contexts (30k+ tokens)*: Speeds up generation. Since the quantized cache transfers 40% of the bytes of FP16, it overcomes memory-bandwidth bottlenecks.*Accuracy cost*: Negligible perplexity increase (+0.54%).\n\nStandard prefix caching fails in multi-turn agent/tool loops when prompts are reconstructed and resent. `q36_server`\n\nprovides a **State Checkpoint Cache** to offload recurrent states and KV pages:\n\n**Mechanism**: At each prompt end, the server snapshots the live engine state (attention KV + the ~63MB recurrent SSM state) to system DRAM. If the next prompt shares a prefix, the state is restored from DRAM in**3ms** instead of triggering a multi-second re-prefill.**Configuration Flags**:`--cache-ram MB`\n\n: Memory allocated for DRAM cache (defaults to auto: ~4 full-context checkpoints, capped at 25% of system RAM, floor 4GB).`--cache-min tokens`\n\n: Minimum prefix token length to trigger checkpointing (default:`2048`\n\n).`--cache-dir path`\n\n: Local directory path to write evicted checkpoints to disk (LRU tiering). Checkpoints persisted here survive server restarts.`--no-state-cache`\n\n: Disables checkpoint caching completely.\n\nThe underlying `q36`\n\nengine supports high-throughput, multi-tenant execution using **continuous batching** and **slot management** primitives. While the OpenAI HTTP server (`q36_server`\n\n) currently processes requests sequentially (on Slot 0), the core library exposes a full multi-tenant scheduler API:\n\n`q36_engine_create_mt(model, max_ctx, n_slots)`\n\n: Instantiates an engine with`n_slots`\n\nconcurrent sequence states. Attention KV cache pages and recurrent SSM state blocks are separate per slot.`q36_engine_prefill_slot(engine, slot, tokens, len, pos0)`\n\n: Processes prompts into a specific slot.`q36_engine_step_active(engine, n_active, active_tokens, positions, out_tokens)`\n\n: Performs a single batched decode step for all`n_active`\n\nslots. To avoid graph launch overheads, active slots are padded/bucketed to`{8, 16, 32, 48, 64}`\n\nto reuse captured CUDA graphs.`q36_engine_slot_move(engine, dst_slot, src_slot)`\n\n: Instantly relocates a slot's entire history (attention KV, SSM state, convolution buffers) in memory (~40us overhead). This allows**slot compaction** upon request eviction to maintain a contiguous active slot block.\n\nThe engine dynamically switches decoding strategies based on the active batch size:\n\n**GEMV Mode (Batch < 16)**: Lowest latency; serial state updates.** Batch-Tiled Mode (Batch >= 16)**: Uses tensor cores and tiled GEMM kernels to load and share weight matrices across all active sequences, achieving aggregate throughput of up to**1,653 tokens/sec** at Batch=64. The engine automatically transitions between these modes, which can also be forced via`Q36_MT_GEMV`\n\n/`Q36_MT_TILED`\n\n.\n\nYou can simulate staggered request arrival, execution, and eviction (utilizing slot compaction under churn) via the benchmark tool:\n\n```\n# Run a continuous batching simulation with up to 48 concurrent slots:\nQ36_CB=48 Q36_CB_NREQ=256 ./q36_bench -m /path/to/model.gguf\n```\n\n[q36_engine.cu](/ambud/q36/blob/main/q36_engine.cu): Core engine loop, CUDA graph setup, Blackwell W4A8 MMA kernels, and FlashAttention.[q36_dequant.cuh](/ambud/q36/blob/main/q36_dequant.cuh): Quantization formats, layouts, and CPU-parallelized dequantization.[tokenizer.c](/ambud/q36/blob/main/tokenizer.c): Native BPE tokenizer implementation.[server.c](/ambud/q36/blob/main/server.c): Zero-dependency OpenAI-compatible HTTP server.\n\n[ARCHITECTURE.md](/ambud/q36/blob/main/docs/ARCHITECTURE.md): Systems deep dive — decode roofline analysis, Blackwell block-scaled MMA, the SSM/attention caching asymmetry, multi-GPU trade-offs.[ENGINE.md](/ambud/q36/blob/main/docs/ENGINE.md): Detailed engine reference — benchmarks, CLI/server flags, project layout.\n\nThis project is licensed under the [AGPL-3.0 License](/ambud/q36/blob/main/LICENSE). If you run a modified version of these engines as a network service, you must make your modified source available to its users.\n\nCopyright is retained by [Ambud Sharma](https://github.com/ambud). The AGPL applies to everyone else's use of this code; Ambud Sharma reserves the right to offer this software under other license terms (dual licensing).\n\nContributions require the [Contributor License Agreement](/ambud/q36/blob/main/CLA.md) (see [CONTRIBUTING.md](/ambud/q36/blob/main/CONTRIBUTING.md)), preserving the project's ability to dual-license.\n\nThis codebase contains third-party components adapted from ** llama.cpp** (licensed under the\n\n**MIT License**):\n\n- Dequantization constants, block layout structures, and scaling logic in\n[q36_dequant.cuh](/ambud/q36/blob/main/q36_dequant.cuh). - Perplexity calculation windowing strategies in\n[ppl.c](/ambud/q36/blob/main/ppl.c).\n\nThe full MIT License text and copyright notices for these components are preserved in the respective source files.\n\nBuilt by [Ambud Sharma](https://github.com/ambud) — connect with me on [LinkedIn](https://www.linkedin.com/in/ambud/).", "url": "https://wpnews.pro/news/performant-c-cuda-inference-engine-for-qwen-3-6-35b-on-rtx-5090-blackwell", "canonical_source": "https://github.com/ambud/q36", "published_at": "2026-07-13 00:13:20+00:00", "updated_at": "2026-07-13 00:34:59.407641+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "ai-products"], "entities": ["Qwen", "NVIDIA", "RTX 5090", "Blackwell", "CUDA", "Hugging Face", "llama.cpp", "vLLM"], "alternates": {"html": "https://wpnews.pro/news/performant-c-cuda-inference-engine-for-qwen-3-6-35b-on-rtx-5090-blackwell", "markdown": "https://wpnews.pro/news/performant-c-cuda-inference-engine-for-qwen-3-6-35b-on-rtx-5090-blackwell.md", "text": "https://wpnews.pro/news/performant-c-cuda-inference-engine-for-qwen-3-6-35b-on-rtx-5090-blackwell.txt", "jsonld": "https://wpnews.pro/news/performant-c-cuda-inference-engine-for-qwen-3-6-35b-on-rtx-5090-blackwell.jsonld"}}