# I got Qwen3.8-27B running on dual RTX 3090s (no NVLink) under WSL2 — every pitfall I hit > Source: > Published: 2026-08-18 09:02:23+00:00 Note: English translation assisted by an LLM. I'm not a native English speaker; all technical content, commands, and measurements come from my own setup. I spent a full week getting Qwen3.8-27B (hybrid GDN architecture, 48 linear-attention + 16 full-attention layers, built-in MTP head) running on my dual RTX 3090 (24GB × 2) box — no NVLink, PCIe Gen4, Windows 10 + WSL2 (Ubuntu-24.04). I hit basically every trap in the book. This is the complete record: what broke, why, and the config that finally gave me 170-210 tok/s on code/JSON. | Item | Configuration | |---|---| | GPU | 2× NVIDIA RTX 3090 (24GB GDDR6X, SM86/Ampere) | | GPU interconnect | No NVLink (PCIe Gen4 x8 bridge) | | CPU / RAM | Ryzen 9 5950X / 64GB DDR4 | | OS | Windows 10 + WSL2 (Ubuntu-24.04) | | Inference engine | SGLang 0.5.17 | | CUDA toolchain | CUDA 13.0 (this exact version matters, see Pitfall 2) | | Model | Qwen3.8-27B-AWQ-INT4 (cyankiwi quant, ~21GB) | | Speculative decoding | DSpark (1.4B draft model) | | Context length | 245,760 tokens | vLLM was the obvious first try. It did not go well: `libnvrtc.so.13` (nvidia-cuda-nvrtc was a 0.0.0a0 placeholder package). After force-reinstalling, flashinfer's sampling JIT demands the CUDA-13-only `--host-stub-linkage-explicit` while local nvcc is 12.8. Engine init crashes every time.**Verdict: skip vLLM 0.27+ for Qwen3.8 on 3090s. The old Docker 0.25.1 works but gives no speedup.** Switching to SGLang was worse: the model froze during weight loading — process hung, port never listened. I suspected parameters and tried dozens of combinations. Useless. Comparing keys in `model.safetensors.index.json` finally revealed the root cause: cyankiwi's Qwen3.8-AWQ-INT4 is a "half-quantized" checkpoint— the MTP head and some linear_attn layers use plain`weight` instead of`weight_packed` , unlike Qwen3.6's fully-quantized checkpoints. Same architecture config: Qwen3.6 runs, Qwen3.8 dies on load. The real fix was toolchain matching: | CUDA | Result | |---|---| | 12.8 | weight loading hangs | | 13.3 | `__NV_ATOMIC_RELAXED` macro removed → verify-graph compilation fails | 13.0 ✅ | perfect match for torch cu130; keeps the cccl macros flashinfer 0.6.15 needs | Plus the no-NVLink triple fix: ``` --disable-custom-all-reduce # avoid "peer access is not supported" --mm-feature-transport cpu # bypass pidfd_getfd crash in CUDA IPC under WSL NCCL_P2P_DISABLE=1 # must disable P2P without NVLink ``` And a SymmMem patch: set `MultimemAllGatherer.enabled=False` in `logits_processor.py` to bypass the SIGFPE from `torch.distributed._symmetric_memory.rendezvous()` on 3090s. It ran! At **~10 tok/s**. Unusable. | Approach | Acceptance | Speed | Verdict | |---|---|---|---| | No speculation (baseline) | — | ~63 tok/s | start | | MTP/EAGLE (built-in head) | 1.07-1.27 | ~37 tok/s | ❌ negative — INT4 quantization damaged the BF16 head | | NGRAM | — | ~56 tok/s | ❌ +50% on JSON only, -30% on normal text | | ReplaySSM | 1.45→1.55 | 40→43 tok/s | ❌ negligible | DSpark (1.4B draft model) | 3.7-4.2 | 170-210 tok/s | ✅✅✅ | DSpark was the only thing that worked. An independent 1.4B draft model proposes a 7-token candidate block per step; the main model verifies the block in one pass. Acceptance of 3.7-4.2 means ~4 tokens accepted per verification — 3x better than MTP on this quantized model. **Symptom**: service starts, but `server_args=` log shows default values for everything — `mm_feature_transport='cuda_ipc'` , `speculative_algorithm=None` . **Cause**: a trailing `\` went missing in the launch script; bash truncates the command there and **silently discards every parameter after it**. The service boots with bare defaults, which crashes on WSL without NVLink. **Lesson**: after editing the launch script, verify with `grep server_args=` that your parameters actually took effect before tuning anything. This may be the truth behind 80% of "my parameters don't work" reports. Qwen3.8's GDN kernels need CUDA-13 compilation. 12.8 hangs on load, 13.3 fails to compile, 13.0 is exactly right. Check which cu version your SGLang torch was built against (cu130), then match the toolchain exactly. `--enable-torch-compile` crashes on GDN (`launcher() missing '_grid_2'` — inductor can't compile `causal_conv1d` ).Qwen3.8's built-in MTP head is BF16, but this INT4 quant quantized the head too — it misjudges on INT4 hidden states, acceptance never rises (1.07-1.27). Not a parameter problem; a quantization problem. Either use a W4A16 model that keeps the BF16 head (limited gain without P2P) or go DSpark. `--mamba-radix-cache-strategy extra_buffer` (recommended to avoid GDN degradation): `--chunked-prefill-size 2048` (recommended for concurrency): for single-request + 50-90K token agent contexts it quadruples prefill iterations and slows TTFT. Test your own workload before copying official parameters. `--mem-fraction-static` too high + large context → no headroom for CUDA graph capture → **verify-graph hangs** (log stalls at `Capturing batches 0%` , GPU 100% but CPU time frozen). Fix: **0.8** (DSpark also needs VRAM for the 1.4B draft), or `--cuda-graph-backend-prefill disabled` . **How to tell a hang from progress**: `ps -o pid,pcpu,time` — CPU TIME growing = healthy, frozen = deadlock. Don't kill the process prematurely; graph capture legitimately takes 90-120s. `--served-model-name` must be a short name Clients send short model names; if the server only exposes the full path → **detokenizer hangs**, health checks fail, everything times out. Add `--served-model-name ` . I had 6 launch entry points (.bashrc, systemd, watchdog, Windows VBS chain…). Editing parameters in one script while a watchdog pulls from another → old config "resurrects" after reboot. Audit all of them: ``` grep -r "sglang\|start-models" ~/.bashrc ~/.wsl-hermes/ /etc/systemd/ ``` and point every entry point at the same config file. ``` python -m sglang.launch_server \ --model-path /home/user2222/models/cyankiwiQwen3.8-27B-AWQ-INT4 \ --served-model-name cyankiwiQwen3.8-27B-AWQ-INT4 \ --port 9090 --tp-size 2 \ --quantization compressed-tensors \ --mem-fraction-static 0.8 \ --kv-cache-dtype fp8_e4m3 \ --chunked-prefill-size 8192 \ --context-length 245760 \ --dtype bfloat16 --mamba-ssm-dtype bfloat16 \ --disable-custom-all-reduce \ --enable-tf32-matmul --schedule-policy lpm \ --trust-remote-code \ --speculative-algorithm DSPARK \ --speculative-draft-model-path /home/user2222/models/Qwen3.8-27B-DSpark \ --speculative-dspark-block-size 7 \ --speculative-draft-model-quantization unquant \ --tool-call-parser qwen3_coder --reasoning-parser qwen3 \ --max-running-requests 2 --allow-auto-truncate \ --cuda-graph-bs-decode 1 2 3 4 5 \ --cuda-graph-backend-prefill disabled \ --mm-feature-transport cpu --stream-interval 1 ``` Official-style benchmarks (DSpark vs no speculation): | Task | No speculation | DSpark | Speedup | |---|---|---|---| | Code/JSON | ~63 | 170-210 tok/s | 3x | | Deep reasoning | ~63 | 153 tok/s | 2.4x | | Greedy sampling | ~63 | 215 tok/s | 3.4x | | Plain text | ~63 | 57 tok/s | ≈parity | | Long context | ~63 | 61 tok/s | ≈parity | | Speculative acceptance | — | 3.7-4.2 | 3x MTP | End-to-end on real tasks (includes prefill + thinking tokens, so below pure decode peak): | Task | Time | Output tokens | End-to-end | |---|---|---|---| | Code (quicksort) | 11.6s | 800 | ~69 tok/s | | JSON structured output | 4.2s | 263 | ~63 tok/s | | Prose | 6.2s | 298 | ~48 tok/s | | Math reasoning | 6.3s | 439 | ~69 tok/s | Stability: 245,760 context, GPU0 23.9GB / GPU1 21.3GB, running 24/7 as an agent backend (code, JSON, tool calls, deep reasoning) with no crashes. Happy to answer questions in the comments — this took me a week and I'd rather you skip it. *Originally written for DigitalMarket.World (World Digital Economy Network). All data from real measurements on my own hardware. Free to share with credit.*