# Local LLM Setup Guide — Windows 11 + WSL2 + Bionic + Qwen Code

> Source: <https://gist.github.com/periman/3c01b374f19eb1b1dbe11bfcdf04c76a>
> Published: 2026-10-05 16:15:09+00:00

**Key takeaway:** On an 8GB RTX 3070 laptop, running Qwen2.5-Coder-14B with ~29 GPU-offloaded layers, Q8_0 KV cache quantization, Flash Attention and a 32768 context length avoids the CPU fallback and request rejections caused by speculative decoding without a draft model and a 4096 context.

Here is the definitive, concise record of our troubleshooting journey. Save this as a reference guide (e.g., `LOCAL_LLM_SETUP_GUIDE.md`). It is structured so that any newcomer, AI instance, or developer can understand the exact architectural pitfalls of running local LLMs on a Windows + WSL2 + Bionic (LM Studio) setup.

**Hardware Profile:** ASUS Laptop, AMD Ryzen 9 5900HX, NVIDIA RTX 3070 Laptop (8GB VRAM), 64GB RAM, Windows 11, WSL2 (Ubuntu).

- **Backend:** Bionic (LM Studio fork) running on Windows.
- **Frontend:** Qwen Code CLI running inside WSL2.
- **Inference:**`llama.cpp` with CUDA 12 runtime.

- **Cause:** WSL2 dynamically reserves ports on Windows startup. Port`1234` is frequently stolen by Windows services (`svchost` ,`winnat` ).
- **Fix:****Never use port 1234.** Change Bionic's API port to`8000` . Ports in the 8000 range are ignored by WSL2's dynamic reservation.
- **Action:** In Bionic:`Local Model API` ->`Edit port` ->`8000` .

- **Cause:**`localhost` inside WSL2 is not the same as`localhost` on Windows. WSL2 cannot see services bound to Windows`127.0.0.1` .
- **Fix A (Recommended):** Enable mirrored networking. Create`C:\Users\periman\.wslconfig` and add:
Then run

```
[wsl2]
networkingMode=mirrored
```

 `wsl --shutdown` in PowerShell (Admin).
- **Fix B (Alternative):** Bind Bionic to all interfaces (`--bind 0.0.0.0` ) and use the Windows host IP (found in`/etc/resolv.conf` ) in your WSL scripts.

- **Cause:** Speculative Decoding is enabled in the UI, but no draft model is selected. This causes Bionic to silently crash the model load, resulting in**0% GPU usage** and the model running entirely on CPU.
- **Fix:** Turn**Speculative Decoding OFF** for all models unless you have a compatible draft model configured.
- **Draft Model Bug:** If you try to use a draft model, avoid`Q8_0` quantized draft models on CUDA 12. They will crash with`error loading model: invalid vector subscript` . Use`Q4_K_M` for drafts.

- **Cause 1:** Running a 30B model on an 8GB GPU. Even with MoE CPU offloading (`--n-cpu-moe` ), the CPU prefill bottleneck makes it take 2.5 minutes to answer a simple prompt.
- **Cause 2:** Running Bionic from an**External HDD** . Windows blocks CUDA initialization from external drives, forcing CPU-only inference.
- **Fix:**  1. Move Bionic to the internal `C:` drive.
  2. Use a **14B model** (e.g.,`Qwen2.5-Coder-14B-Instruct` ).
  3. Set `GPU Offload` to the maximum number of layers that fits in ~7.5GB VRAM (usually ~29 layers).
  4. Set `K Cache Quantization` and`V Cache Quantization` to`Q8_0` .
  5. Enable `Flash Attention` and`Unified KV Cache` .
- Move Bionic to the internal 

- **Cause:** Qwen Code uses an environment variable`QWEN_HOME` pointing to a specific directory (`/home/periman/qwen-local-agent/qwen-home` ). Editing`~/.qwen/settings.json` does nothing.
- **Fix:**  1. Check `printenv | grep QWEN` .
  2. Edit the actual `settings.json` inside the`QWEN_HOME` directory.
  3. Ensure the `"id"` in`modelProviders`**exactly matches** the model ID exposed by Bionic's API (check with`curl -s http://localhost:8000/v1/models` ).
  4. Ensure `"baseUrl"` is set to`http://localhost:8000/v1` .
- Check 

- **Cause:** Qwen Code sends a massive ~20,000 token system prompt. If Bionic's model context is set to 4096, it rejects the request.
- **Fix:** Set Bionic's`Context Length` to`32768` .

- **Model:** Qwen2.5-Coder-14B-Instruct (or Q3_K_M for even more GPU offload).
- **GPU Offload:** 29 layers (monitor Task Manager`GPU 1` , keep VRAM under 7.8GB).
- **Context Length:** 32768.
- **K Cache Quantization:** Q8_0.
- **V Cache Quantization:** Q8_0.
- **Flash Attention:** ON.
- **Unified KV Cache:** ON.
- **Speculative Decoding:** OFF.

- **Environment Variables:**`OPENAI_API_KEY="lm-studio"` ,`QWEN_HOME=/home/periman/qwen-local-agent/qwen-home` .
- **Qwen Code Update:** Run`sudo npm install -g @qwen-code/qwen-code@latest` . If`Ripgrep EACCES` error occurs, run`sudo chmod +x /usr/lib/node_modules/@qwen-code/qwen-code/vendor/ripgrep/x64-linux/rg` .

1. Put Bionic on your `C:` drive.
2. Use port `8000` .
3. Use a 14B model, not 30B.
4. Turn `Speculative Decoding` OFF.
5. Find the real `settings.json` using`printenv | grep QWEN` .
6. Match the model ID in the config exactly to Bionic's API output.
