Note: English translation assisted by an LLM. I'm not a native English speaker; all technical content, commands, and measurements come from my own setup.
I spent a full week getting Qwen3.8-27B (hybrid GDN architecture, 48 linear-attention + 16 full-attention layers, built-in MTP head) running on my dual RTX 3090 (24GB × 2) box — no NVLink, PCIe Gen4, Windows 10 + WSL2 (Ubuntu-24.04). I hit basically every trap in the book. This is the complete record: what broke, why, and the config that finally gave me 170-210 tok/s on code/JSON.
| Item | Configuration |
|---|---|
| GPU | 2× NVIDIA RTX 3090 (24GB GDDR6X, SM86/Ampere) |
| GPU interconnect | |
| No NVLink (PCIe Gen4 x8 bridge) | |
| CPU / RAM | Ryzen 9 5950X / 64GB DDR4 |
| OS | Windows 10 + WSL2 (Ubuntu-24.04) |
| Inference engine | SGLang 0.5.17 |
| CUDA toolchain | |
| CUDA 13.0 (this exact version matters, see Pitfall 2) | |
| Model | Qwen3.8-27B-AWQ-INT4 (cyankiwi quant, ~21GB) |
| Speculative decoding | |
| DSpark (1.4B draft model) | |
| Context length | 245,760 tokens |
vLLM was the obvious first try. It did not go well:
libnvrtc.so.13
(nvidia-cuda-nvrtc was a 0.0.0a0 placeholder package). After force-reinstalling, flashinfer's sampling JIT demands the CUDA-13-only --host-stub-linkage-explicit
while local nvcc is 12.8. Engine init crashes every time.Verdict: skip vLLM 0.27+ for Qwen3.8 on 3090s. The old Docker 0.25.1 works but gives no speedup.
Switching to SGLang was worse: the model froze during weight — process hung, port never listened.
I suspected parameters and tried dozens of combinations. Useless. Comparing keys in model.safetensors.index.json
finally revealed the root cause:
cyankiwi's Qwen3.8-AWQ-INT4 is a "half-quantized" checkpoint— the MTP head and some linear_attn layers use plainweight
instead ofweight_packed
, unlike Qwen3.6's fully-quantized checkpoints. Same architecture config: Qwen3.6 runs, Qwen3.8 dies on load.
The real fix was toolchain matching:
| CUDA | Result |
|---|---|
| 12.8 | weight hangs |
| 13.3 | |
__NV_ATOMIC_RELAXED macro removed → verify-graph compilation fails |
|
| 13.0 ✅ | |
| perfect match for torch cu130; keeps the cccl macros flashinfer 0.6.15 needs |
Plus the no-NVLink triple fix:
--disable-custom-all-reduce # avoid "peer access is not supported"
--mm-feature-transport cpu # bypass pidfd_getfd crash in CUDA IPC under WSL
NCCL_P2P_DISABLE=1 # must disable P2P without NVLink
And a SymmMem patch: set MultimemAllGatherer.enabled=False
in logits_processor.py
to bypass the SIGFPE from torch.distributed._symmetric_memory.rendezvous()
on 3090s.
It ran! At ~10 tok/s. Unusable.
| Approach | Acceptance | Speed | Verdict |
|---|---|---|---|
| No speculation (baseline) | — | ~63 tok/s | start |
| MTP/EAGLE (built-in head) | 1.07-1.27 | ~37 tok/s | ❌ negative — INT4 quantization damaged the BF16 head |
| NGRAM | — | ~56 tok/s | ❌ +50% on JSON only, -30% on normal text |
| ReplaySSM | 1.45→1.55 | 40→43 tok/s | ❌ negligible |
| DSpark (1.4B draft model) | |||
| 3.7-4.2 | |||
| 170-210 tok/s | |||
| ✅✅✅ |
DSpark was the only thing that worked. An independent 1.4B draft model proposes a 7-token candidate block per step; the main model verifies the block in one pass. Acceptance of 3.7-4.2 means ~4 tokens accepted per verification — 3x better than MTP on this quantized model.
Symptom: service starts, but server_args=
log shows default values for everything — mm_feature_transport='cuda_ipc'
, speculative_algorithm=None
.
Cause: a trailing \
went missing in the launch script; bash truncates the command there and silently discards every parameter after it. The service boots with bare defaults, which crashes on WSL without NVLink.
Lesson: after editing the launch script, verify with grep server_args=
that your parameters actually took effect before tuning anything. This may be the truth behind 80% of "my parameters don't work" reports.
Qwen3.8's GDN kernels need CUDA-13 compilation. 12.8 hangs on load, 13.3 fails to compile, 13.0 is exactly right. Check which cu version your SGLang torch was built against (cu130), then match the toolchain exactly.
--enable-torch-compile
crashes on GDN (launcher() missing '_grid_2'
— inductor can't compile causal_conv1d
).Qwen3.8's built-in MTP head is BF16, but this INT4 quant quantized the head too — it misjudges on INT4 hidden states, acceptance never rises (1.07-1.27). Not a parameter problem; a quantization problem. Either use a W4A16 model that keeps the BF16 head (limited gain without P2P) or go DSpark.
--mamba-radix-cache-strategy extra_buffer
(recommended to avoid GDN degradation): --chunked-prefill-size 2048
(recommended for concurrency): for single-request + 50-90K token agent contexts it quadruples prefill iterations and slows TTFT. Test your own workload before copying official parameters.
--mem-fraction-static
too high + large context → no headroom for CUDA graph capture → verify-graph hangs (log stalls at Capturing batches 0%
, GPU 100% but CPU time frozen). Fix: 0.8 (DSpark also needs VRAM for the 1.4B draft), or --cuda-graph-backend-prefill disabled
.
How to tell a hang from progress: ps -o pid,pcpu,time
— CPU TIME growing = healthy, frozen = deadlock. Don't kill the process prematurely; graph capture legitimately takes 90-120s.
--served-model-name
must be a short name
Clients send short model names; if the server only exposes the full path → detokenizer hangs, health checks fail, everything times out. Add --served-model-name <short-name>
.
I had 6 launch entry points (.bashrc, systemd, watchdog, Windows VBS chain…). Editing parameters in one script while a watchdog pulls from another → old config "resurrects" after reboot. Audit all of them:
grep -r "sglang\|start-models" ~/.bashrc ~/.wsl-hermes/ /etc/systemd/
and point every entry point at the same config file.
python -m sglang.launch_server \
--model-path /home/user2222/models/cyankiwiQwen3.8-27B-AWQ-INT4 \
--served-model-name cyankiwiQwen3.8-27B-AWQ-INT4 \
--port 9090 --tp-size 2 \
--quantization compressed-tensors \
--mem-fraction-static 0.8 \
--kv-cache-dtype fp8_e4m3 \
--chunked-prefill-size 8192 \
--context-length 245760 \
--dtype bfloat16 --mamba-ssm-dtype bfloat16 \
--disable-custom-all-reduce \
--enable-tf32-matmul --schedule-policy lpm \
--trust-remote-code \
--speculative-algorithm DSPARK \
--speculative-draft-model-path /home/user2222/models/Qwen3.8-27B-DSpark \
--speculative-dspark-block-size 7 \
--speculative-draft-model-quantization unquant \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--max-running-requests 2 --allow-auto-truncate \
--cuda-graph-bs-decode 1 2 3 4 5 \
--cuda-graph-backend-prefill disabled \
--mm-feature-transport cpu --stream-interval 1
Official-style benchmarks (DSpark vs no speculation):
| Task | No speculation | DSpark | Speedup |
|---|---|---|---|
| Code/JSON | ~63 | 170-210 tok/s | |
| 3x | |||
| Deep reasoning | ~63 | 153 tok/s | |
| 2.4x | |||
| Greedy sampling | ~63 | 215 tok/s | |
| 3.4x | |||
| Plain text | ~63 | 57 tok/s | ≈parity |
| Long context | ~63 | 61 tok/s | ≈parity |
| Speculative acceptance | — | 3.7-4.2 | 3x MTP |
End-to-end on real tasks (includes prefill + thinking tokens, so below pure decode peak):
| Task | Time | Output tokens | End-to-end |
|---|---|---|---|
| Code (quicksort) | 11.6s | 800 | ~69 tok/s |
| JSON structured output | 4.2s | 263 | ~63 tok/s |
| Prose | 6.2s | 298 | ~48 tok/s |
| Math reasoning | 6.3s | 439 | ~69 tok/s |
Stability: 245,760 context, GPU0 23.9GB / GPU1 21.3GB, running 24/7 as an agent backend (code, JSON, tool calls, deep reasoning) with no crashes.
Happy to answer questions in the comments — this took me a week and I'd rather you skip it.
Originally written for DigitalMarket.World (World Digital Economy Network). All data from real measurements on my own hardware. Free to share with credit.