{"slug": "i-got-qwen3-8-27b-running-on-dual-rtx-3090s-no-nvlink-under-wsl2-every-pitfall-i", "title": "I got Qwen3.8-27B running on dual RTX 3090s (no NVLink) under WSL2 — every pitfall I hit", "summary": "A developer detailed the process of running Qwen3.8-27B on dual RTX 3090s without NVLink under WSL2, achieving 170-210 tok/s on code/JSON. The setup required specific CUDA 13.0 toolchain, SGLang 0.5.17, and DSpark speculative decoding, overcoming pitfalls like vLLM incompatibility and WSL-specific crashes.", "body_md": "Note: English translation assisted by an LLM. I'm not a native English speaker; all technical content, commands, and measurements come from my own setup.\n\nI spent a full week getting Qwen3.8-27B (hybrid GDN architecture, 48 linear-attention + 16 full-attention layers, built-in MTP head) running on my dual RTX 3090 (24GB × 2) box — no NVLink, PCIe Gen4, Windows 10 + WSL2 (Ubuntu-24.04). I hit basically every trap in the book. This is the complete record: what broke, why, and the config that finally gave me 170-210 tok/s on code/JSON.\n\n| Item | Configuration |\n|---|---|\n| GPU | 2× NVIDIA RTX 3090 (24GB GDDR6X, SM86/Ampere) |\n| GPU interconnect |\nNo NVLink (PCIe Gen4 x8 bridge) |\n| CPU / RAM | Ryzen 9 5950X / 64GB DDR4 |\n| OS | Windows 10 + WSL2 (Ubuntu-24.04) |\n| Inference engine | SGLang 0.5.17 |\n| CUDA toolchain |\nCUDA 13.0 (this exact version matters, see Pitfall 2) |\n| Model | Qwen3.8-27B-AWQ-INT4 (cyankiwi quant, ~21GB) |\n| Speculative decoding |\nDSpark (1.4B draft model) |\n| Context length | 245,760 tokens |\n\nvLLM was the obvious first try. It did not go well:\n\n`libnvrtc.so.13`\n\n(nvidia-cuda-nvrtc was a 0.0.0a0 placeholder package). After force-reinstalling, flashinfer's sampling JIT demands the CUDA-13-only `--host-stub-linkage-explicit`\n\nwhile local nvcc is 12.8. Engine init crashes every time.**Verdict: skip vLLM 0.27+ for Qwen3.8 on 3090s. The old Docker 0.25.1 works but gives no speedup.**\n\nSwitching to SGLang was worse: the model froze during weight loading — process hung, port never listened.\n\nI suspected parameters and tried dozens of combinations. Useless. Comparing keys in `model.safetensors.index.json`\n\nfinally revealed the root cause:\n\ncyankiwi's Qwen3.8-AWQ-INT4 is a \"half-quantized\" checkpoint— the MTP head and some linear_attn layers use plain`weight`\n\ninstead of`weight_packed`\n\n, unlike Qwen3.6's fully-quantized checkpoints. Same architecture config: Qwen3.6 runs, Qwen3.8 dies on load.\n\nThe real fix was toolchain matching:\n\n| CUDA | Result |\n|---|---|\n| 12.8 | weight loading hangs |\n| 13.3 |\n`__NV_ATOMIC_RELAXED` macro removed → verify-graph compilation fails |\n13.0 ✅ |\nperfect match for torch cu130; keeps the cccl macros flashinfer 0.6.15 needs |\n\nPlus the no-NVLink triple fix:\n\n```\n--disable-custom-all-reduce     # avoid \"peer access is not supported\"\n--mm-feature-transport cpu      # bypass pidfd_getfd crash in CUDA IPC under WSL\nNCCL_P2P_DISABLE=1              # must disable P2P without NVLink\n```\n\nAnd a SymmMem patch: set `MultimemAllGatherer.enabled=False`\n\nin `logits_processor.py`\n\nto bypass the SIGFPE from `torch.distributed._symmetric_memory.rendezvous()`\n\non 3090s.\n\nIt ran! At **~10 tok/s**. Unusable.\n\n| Approach | Acceptance | Speed | Verdict |\n|---|---|---|---|\n| No speculation (baseline) | — | ~63 tok/s | start |\n| MTP/EAGLE (built-in head) | 1.07-1.27 | ~37 tok/s | ❌ negative — INT4 quantization damaged the BF16 head |\n| NGRAM | — | ~56 tok/s | ❌ +50% on JSON only, -30% on normal text |\n| ReplaySSM | 1.45→1.55 | 40→43 tok/s | ❌ negligible |\nDSpark (1.4B draft model) |\n3.7-4.2 |\n170-210 tok/s |\n✅✅✅ |\n\nDSpark was the only thing that worked. An independent 1.4B draft model proposes a 7-token candidate block per step; the main model verifies the block in one pass. Acceptance of 3.7-4.2 means ~4 tokens accepted per verification — 3x better than MTP on this quantized model.\n\n**Symptom**: service starts, but `server_args=`\n\nlog shows default values for everything — `mm_feature_transport='cuda_ipc'`\n\n, `speculative_algorithm=None`\n\n.\n\n**Cause**: a trailing `\\`\n\nwent missing in the launch script; bash truncates the command there and **silently discards every parameter after it**. The service boots with bare defaults, which crashes on WSL without NVLink.\n\n**Lesson**: after editing the launch script, verify with `grep server_args=`\n\nthat your parameters actually took effect before tuning anything. This may be the truth behind 80% of \"my parameters don't work\" reports.\n\nQwen3.8's GDN kernels need CUDA-13 compilation. 12.8 hangs on load, 13.3 fails to compile, 13.0 is exactly right. Check which cu version your SGLang torch was built against (cu130), then match the toolchain exactly.\n\n`--enable-torch-compile`\n\ncrashes on GDN (`launcher() missing '_grid_2'`\n\n— inductor can't compile `causal_conv1d`\n\n).Qwen3.8's built-in MTP head is BF16, but this INT4 quant quantized the head too — it misjudges on INT4 hidden states, acceptance never rises (1.07-1.27). Not a parameter problem; a quantization problem. Either use a W4A16 model that keeps the BF16 head (limited gain without P2P) or go DSpark.\n\n`--mamba-radix-cache-strategy extra_buffer`\n\n(recommended to avoid GDN degradation): `--chunked-prefill-size 2048`\n\n(recommended for concurrency): for single-request + 50-90K token agent contexts it quadruples prefill iterations and slows TTFT. Test your own workload before copying official parameters.\n\n`--mem-fraction-static`\n\ntoo high + large context → no headroom for CUDA graph capture → **verify-graph hangs** (log stalls at `Capturing batches 0%`\n\n, GPU 100% but CPU time frozen). Fix: **0.8** (DSpark also needs VRAM for the 1.4B draft), or `--cuda-graph-backend-prefill disabled`\n\n.\n\n**How to tell a hang from progress**: `ps -o pid,pcpu,time`\n\n— CPU TIME growing = healthy, frozen = deadlock. Don't kill the process prematurely; graph capture legitimately takes 90-120s.\n\n`--served-model-name`\n\nmust be a short name\nClients send short model names; if the server only exposes the full path → **detokenizer hangs**, health checks fail, everything times out. Add `--served-model-name <short-name>`\n\n.\n\nI had 6 launch entry points (.bashrc, systemd, watchdog, Windows VBS chain…). Editing parameters in one script while a watchdog pulls from another → old config \"resurrects\" after reboot. Audit all of them:\n\n```\ngrep -r \"sglang\\|start-models\" ~/.bashrc ~/.wsl-hermes/ /etc/systemd/\n```\n\nand point every entry point at the same config file.\n\n```\npython -m sglang.launch_server \\\n  --model-path /home/user2222/models/cyankiwiQwen3.8-27B-AWQ-INT4 \\\n  --served-model-name cyankiwiQwen3.8-27B-AWQ-INT4 \\\n  --port 9090 --tp-size 2 \\\n  --quantization compressed-tensors \\\n  --mem-fraction-static 0.8 \\\n  --kv-cache-dtype fp8_e4m3 \\\n  --chunked-prefill-size 8192 \\\n  --context-length 245760 \\\n  --dtype bfloat16 --mamba-ssm-dtype bfloat16 \\\n  --disable-custom-all-reduce \\\n  --enable-tf32-matmul --schedule-policy lpm \\\n  --trust-remote-code \\\n  --speculative-algorithm DSPARK \\\n  --speculative-draft-model-path /home/user2222/models/Qwen3.8-27B-DSpark \\\n  --speculative-dspark-block-size 7 \\\n  --speculative-draft-model-quantization unquant \\\n  --tool-call-parser qwen3_coder --reasoning-parser qwen3 \\\n  --max-running-requests 2 --allow-auto-truncate \\\n  --cuda-graph-bs-decode 1 2 3 4 5 \\\n  --cuda-graph-backend-prefill disabled \\\n  --mm-feature-transport cpu --stream-interval 1\n```\n\nOfficial-style benchmarks (DSpark vs no speculation):\n\n| Task | No speculation | DSpark | Speedup |\n|---|---|---|---|\n| Code/JSON | ~63 | 170-210 tok/s |\n3x |\n| Deep reasoning | ~63 | 153 tok/s |\n2.4x |\n| Greedy sampling | ~63 | 215 tok/s |\n3.4x |\n| Plain text | ~63 | 57 tok/s | ≈parity |\n| Long context | ~63 | 61 tok/s | ≈parity |\n| Speculative acceptance | — | 3.7-4.2 | 3x MTP |\n\nEnd-to-end on real tasks (includes prefill + thinking tokens, so below pure decode peak):\n\n| Task | Time | Output tokens | End-to-end |\n|---|---|---|---|\n| Code (quicksort) | 11.6s | 800 | ~69 tok/s |\n| JSON structured output | 4.2s | 263 | ~63 tok/s |\n| Prose | 6.2s | 298 | ~48 tok/s |\n| Math reasoning | 6.3s | 439 | ~69 tok/s |\n\nStability: 245,760 context, GPU0 23.9GB / GPU1 21.3GB, running 24/7 as an agent backend (code, JSON, tool calls, deep reasoning) with no crashes.\n\nHappy to answer questions in the comments — this took me a week and I'd rather you skip it.\n\n*Originally written for DigitalMarket.World (World Digital Economy Network). All data from real measurements on my own hardware. Free to share with credit.*", "url": "https://wpnews.pro/news/i-got-qwen3-8-27b-running-on-dual-rtx-3090s-no-nvlink-under-wsl2-every-pitfall-i", "canonical_source": "https://dev.to/digitalmarket-world/i-got-qwen38-27b-running-on-dual-rtx-3090s-no-nvlink-under-wsl2-every-pitfall-i-hit-3oao", "published_at": "2026-08-18 09:02:23+00:00", "updated_at": "2026-08-18 09:12:35.649308+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Qwen3.8-27B", "RTX 3090", "WSL2", "SGLang", "vLLM", "DSpark", "CUDA 13.0", "cyankiwi"], "alternates": {"html": "https://wpnews.pro/news/i-got-qwen3-8-27b-running-on-dual-rtx-3090s-no-nvlink-under-wsl2-every-pitfall-i", "markdown": "https://wpnews.pro/news/i-got-qwen3-8-27b-running-on-dual-rtx-3090s-no-nvlink-under-wsl2-every-pitfall-i.md", "text": "https://wpnews.pro/news/i-got-qwen3-8-27b-running-on-dual-rtx-3090s-no-nvlink-under-wsl2-every-pitfall-i.txt", "jsonld": "https://wpnews.pro/news/i-got-qwen3-8-27b-running-on-dual-rtx-3090s-no-nvlink-under-wsl2-every-pitfall-i.jsonld"}}