A hyper-optimized, zero-dependency C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 / Blackwell.
A zero-dependency C/CUDA codebase specialized for a single model + GPU pairing — Qwen3.6-35B-A3B (MXFP4 GGUF) on RTX 5090 (sm_120a) — on the bet that a dedicated engine beats generic runtimes (llama.cpp / vLLM / SGLang) on their long tail. Status: faster than llama.cpp at every measured point.
- 🚀
Extreme Prefill Throughput: Tensor-core FlashAttention prefill reaching** 13.4k tokens/sec**(at context depth 2,048, measured on a GPU clocked at 400W) and** 7.6k tokens/sec**(at context depth 90k). - ⚡
Fast Decode: Native token generation speeds of** 270+ tokens/secusing captured CUDA graphs with zero host syncs. - 🧠
Hybrid Architecture Support: Native CUDA implementations for both 10 full-attention layers (using W4A8 block-scaled MMA MoE) and 30 recurrent SSM layers (gated-DeltaNet scans). - 💾
Dual-Tier State Management: Zero-overhead VRAM saving ( 2.5× smaller KV cache**via
--kv-quant
) and DRAM/Disk state checkpoint caching (restoring context states in3ms). - 🌐
OpenAI-Compatible Server: A zero-dependency, prefix-cached HTTP server (q36_server
) supporting SSE streaming and tool calling. - 🛠️
Developer Tooling: Built-in benchmark tools (q36_bench
), perplexity evaluation harnesses (q36_ppl
), and CPU-only validation helpers.
GPU: An NVIDIA RTX 5090 or RTX 6000 Pro Blackwell GPU (Blackwell architecture,sm_120a
/ compute capability 12.0+ is required).*Note: The engine is architecturally compatible with the RTX 6000 Pro Blackwell (and benefits from its 96 GB VRAM), but has not yet been physically tested on it.*Software: Linux withCUDA Toolkit 13.1installed — the version the engine is developed and tested on (12.x releases that exposesm_120a
may work but are untested). Ensure the CUDA compiler (nvcc
) is on your systemPATH
.Model: Download the model weights in MXFP4 GGUF format fromHugging Face: unsloth/Qwen3.6-35B-A3B-MTP-GGUF.
Run the following commands from the repository root:
make tools
./q36_info /path/to/model.gguf
make q36 q36_bench q36_server
Interactive CLI: Run the interactive chat shell:
./q36 -m /path/to/model.gguf
OpenAI Server: Start the HTTP API server with state caching enabled:
./q36_server -m /path/to/model.gguf --port 8080 --ctx 32768
Benchmark: Run throughput tests:
./q36_bench -m /path/to/model.gguf
Due to the hybrid model architecture (10 attention layers + 30 recurrent SSM layers), q36
optimizes KV cache and state memory at two levels:
By default, attention keys and values are stored in FP16. Toggle quantization with the --kv-quant
flag:
Mechanism: Quantizes keys to** Q8_0**(8-bit) and values to** MXFP4**(4-bit block-scaled FP4).** VRAM savings**: Reduces active VRAM KV cache size by** 2.5×**, freeing up space for long contexts and larger batches.** Performance trade-offs**:Short contexts: Adds small kernel overhead (~5% slower decoding).Long contexts (30k+ tokens): Speeds up generation. Since the quantized cache transfers 40% of the bytes of FP16, it overcomes memory-bandwidth bottlenecks.Accuracy cost: Negligible perplexity increase (+0.54%).
Standard prefix caching fails in multi-turn agent/tool loops when prompts are reconstructed and resent. q36_server
provides a State Checkpoint Cache to offload recurrent states and KV pages:
Mechanism: At each prompt end, the server snapshots the live engine state (attention KV + the ~63MB recurrent SSM state) to system DRAM. If the next prompt shares a prefix, the state is restored from DRAM in3ms instead of triggering a multi-second re-prefill.Configuration Flags:--cache-ram MB
: Memory allocated for DRAM cache (defaults to auto: ~4 full-context checkpoints, capped at 25% of system RAM, floor 4GB).--cache-min tokens
: Minimum prefix token length to trigger checkpointing (default:2048
).--cache-dir path
: Local directory path to write evicted checkpoints to disk (LRU tiering). Checkpoints persisted here survive server restarts.--no-state-cache
: Disables checkpoint caching completely.
The underlying q36
engine supports high-throughput, multi-tenant execution using continuous batching and slot management primitives. While the OpenAI HTTP server (q36_server
) currently processes requests sequentially (on Slot 0), the core library exposes a full multi-tenant scheduler API:
q36_engine_create_mt(model, max_ctx, n_slots)
: Instantiates an engine withn_slots
concurrent sequence states. Attention KV cache pages and recurrent SSM state blocks are separate per slot.q36_engine_prefill_slot(engine, slot, tokens, len, pos0)
: Processes prompts into a specific slot.q36_engine_step_active(engine, n_active, active_tokens, positions, out_tokens)
: Performs a single batched decode step for alln_active
slots. To avoid graph launch overheads, active slots are padded/bucketed to{8, 16, 32, 48, 64}
to reuse captured CUDA graphs.q36_engine_slot_move(engine, dst_slot, src_slot)
: Instantly relocates a slot's entire history (attention KV, SSM state, convolution buffers) in memory (~40us overhead). This allowsslot compaction upon request eviction to maintain a contiguous active slot block.
The engine dynamically switches decoding strategies based on the active batch size:
GEMV Mode (Batch < 16): Lowest latency; serial state updates.** Batch-Tiled Mode (Batch >= 16): Uses tensor cores and tiled GEMM kernels to load and share weight matrices across all active sequences, achieving aggregate throughput of up to1,653 tokens/sec** at Batch=64. The engine automatically transitions between these modes, which can also be forced viaQ36_MT_GEMV
/Q36_MT_TILED
.
You can simulate staggered request arrival, execution, and eviction (utilizing slot compaction under churn) via the benchmark tool:
Q36_CB=48 Q36_CB_NREQ=256 ./q36_bench -m /path/to/model.gguf
q36_engine.cu: Core engine loop, CUDA graph setup, Blackwell W4A8 MMA kernels, and FlashAttention.q36_dequant.cuh: Quantization formats, layouts, and CPU-parallelized dequantization.tokenizer.c: Native BPE tokenizer implementation.server.c: Zero-dependency OpenAI-compatible HTTP server.
ARCHITECTURE.md: Systems deep dive — decode roofline analysis, Blackwell block-scaled MMA, the SSM/attention caching asymmetry, multi-GPU trade-offs.ENGINE.md: Detailed engine reference — benchmarks, CLI/server flags, project layout.
This project is licensed under the AGPL-3.0 License. If you run a modified version of these engines as a network service, you must make your modified source available to its users.
Copyright is retained by Ambud Sharma. The AGPL applies to everyone else's use of this code; Ambud Sharma reserves the right to offer this software under other license terms (dual licensing).
Contributions require the Contributor License Agreement (see CONTRIBUTING.md), preserving the project's ability to dual-license.
This codebase contains third-party components adapted from ** llama.cpp** (licensed under the
MIT License):
- Dequantization constants, block layout structures, and scaling logic in q36_dequant.cuh. - Perplexity calculation windowing strategies in ppl.c.
The full MIT License text and copyright notices for these components are preserved in the respective source files.
Built by Ambud Sharma — connect with me on LinkedIn.