{"slug": "local-llm-setup-guide-windows-11-wsl2-bionic-qwen-code", "title": "Local LLM Setup Guide — Windows 11 + WSL2 + Bionic + Qwen Code", "summary": "A developer documented a troubleshooting guide for running local LLMs on a Windows 11 + WSL2 setup, pairing the Bionic (LM Studio fork) backend with the Qwen Code CLI and llama.cpp on CUDA 12. The guide identifies concrete pitfalls — WSL2 port 1234 collisions, localhost isolation between WSL2 and Windows, silent CPU fallback from misconfigured speculative decoding, and CUDA failures when running from an external drive — and prescribes fixes including mirrored networking, port 8000, a 14B Qwen2.5-Coder model with ~29 GPU-offloaded layers, and a 32768 context length to accommodate Qwen Code's ~20,000-token system prompt.", "body_md": "Here is the definitive, concise record of our troubleshooting journey. Save this as a reference guide (e.g., `LOCAL_LLM_SETUP_GUIDE.md`). It is structured so that any newcomer, AI instance, or developer can understand the exact architectural pitfalls of running local LLMs on a Windows + WSL2 + Bionic (LM Studio) setup.\n\n**Hardware Profile:** ASUS Laptop, AMD Ryzen 9 5900HX, NVIDIA RTX 3070 Laptop (8GB VRAM), 64GB RAM, Windows 11, WSL2 (Ubuntu).\n\n- **Backend:** Bionic (LM Studio fork) running on Windows.\n- **Frontend:** Qwen Code CLI running inside WSL2.\n- **Inference:**`llama.cpp` with CUDA 12 runtime.\n\n- **Cause:** WSL2 dynamically reserves ports on Windows startup. Port`1234` is frequently stolen by Windows services (`svchost` ,`winnat` ).\n- **Fix:****Never use port 1234.** Change Bionic's API port to`8000` . Ports in the 8000 range are ignored by WSL2's dynamic reservation.\n- **Action:** In Bionic:`Local Model API` ->`Edit port` ->`8000` .\n\n- **Cause:**`localhost` inside WSL2 is not the same as`localhost` on Windows. WSL2 cannot see services bound to Windows`127.0.0.1` .\n- **Fix A (Recommended):** Enable mirrored networking. Create`C:\\Users\\periman\\.wslconfig` and add:\nThen run\n\n```\n[wsl2]\nnetworkingMode=mirrored\n```\n\n `wsl --shutdown` in PowerShell (Admin).\n- **Fix B (Alternative):** Bind Bionic to all interfaces (`--bind 0.0.0.0` ) and use the Windows host IP (found in`/etc/resolv.conf` ) in your WSL scripts.\n\n- **Cause:** Speculative Decoding is enabled in the UI, but no draft model is selected. This causes Bionic to silently crash the model load, resulting in**0% GPU usage** and the model running entirely on CPU.\n- **Fix:** Turn**Speculative Decoding OFF** for all models unless you have a compatible draft model configured.\n- **Draft Model Bug:** If you try to use a draft model, avoid`Q8_0` quantized draft models on CUDA 12. They will crash with`error loading model: invalid vector subscript` . Use`Q4_K_M` for drafts.\n\n- **Cause 1:** Running a 30B model on an 8GB GPU. Even with MoE CPU offloading (`--n-cpu-moe` ), the CPU prefill bottleneck makes it take 2.5 minutes to answer a simple prompt.\n- **Cause 2:** Running Bionic from an**External HDD** . Windows blocks CUDA initialization from external drives, forcing CPU-only inference.\n- **Fix:**  1. Move Bionic to the internal `C:` drive.\n  2. Use a **14B model** (e.g.,`Qwen2.5-Coder-14B-Instruct` ).\n  3. Set `GPU Offload` to the maximum number of layers that fits in ~7.5GB VRAM (usually ~29 layers).\n  4. Set `K Cache Quantization` and`V Cache Quantization` to`Q8_0` .\n  5. Enable `Flash Attention` and`Unified KV Cache` .\n- Move Bionic to the internal \n\n- **Cause:** Qwen Code uses an environment variable`QWEN_HOME` pointing to a specific directory (`/home/periman/qwen-local-agent/qwen-home` ). Editing`~/.qwen/settings.json` does nothing.\n- **Fix:**  1. Check `printenv | grep QWEN` .\n  2. Edit the actual `settings.json` inside the`QWEN_HOME` directory.\n  3. Ensure the `\"id\"` in`modelProviders`**exactly matches** the model ID exposed by Bionic's API (check with`curl -s http://localhost:8000/v1/models` ).\n  4. Ensure `\"baseUrl\"` is set to`http://localhost:8000/v1` .\n- Check \n\n- **Cause:** Qwen Code sends a massive ~20,000 token system prompt. If Bionic's model context is set to 4096, it rejects the request.\n- **Fix:** Set Bionic's`Context Length` to`32768` .\n\n- **Model:** Qwen2.5-Coder-14B-Instruct (or Q3_K_M for even more GPU offload).\n- **GPU Offload:** 29 layers (monitor Task Manager`GPU 1` , keep VRAM under 7.8GB).\n- **Context Length:** 32768.\n- **K Cache Quantization:** Q8_0.\n- **V Cache Quantization:** Q8_0.\n- **Flash Attention:** ON.\n- **Unified KV Cache:** ON.\n- **Speculative Decoding:** OFF.\n\n- **Environment Variables:**`OPENAI_API_KEY=\"lm-studio\"` ,`QWEN_HOME=/home/periman/qwen-local-agent/qwen-home` .\n- **Qwen Code Update:** Run`sudo npm install -g @qwen-code/qwen-code@latest` . If`Ripgrep EACCES` error occurs, run`sudo chmod +x /usr/lib/node_modules/@qwen-code/qwen-code/vendor/ripgrep/x64-linux/rg` .\n\n1. Put Bionic on your `C:` drive.\n2. Use port `8000` .\n3. Use a 14B model, not 30B.\n4. Turn `Speculative Decoding` OFF.\n5. Find the real `settings.json` using`printenv | grep QWEN` .\n6. Match the model ID in the config exactly to Bionic's API output.", "url": "https://wpnews.pro/news/local-llm-setup-guide-windows-11-wsl2-bionic-qwen-code", "canonical_source": "https://gist.github.com/periman/3c01b374f19eb1b1dbe11bfcdf04c76a", "published_at": "2026-10-05 16:15:09+00:00", "updated_at": "2026-10-18 13:19:29.122541+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Qwen Code", "Bionic", "LM Studio", "llama.cpp", "NVIDIA", "WSL2", "Qwen2.5-Coder-14B-Instruct", "Windows 11"], "also_reported_by": [], "takeaway": "On an 8GB RTX 3070 laptop, running Qwen2.5-Coder-14B with ~29 GPU-offloaded layers, Q8_0 KV cache quantization, Flash Attention and a 32768 context length avoids the CPU fallback and request rejections caused by speculative decoding without a draft model and a 4096 context.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/local-llm-setup-guide-windows-11-wsl2-bionic-qwen-code", "markdown": "https://wpnews.pro/news/local-llm-setup-guide-windows-11-wsl2-bionic-qwen-code.md", "text": "https://wpnews.pro/news/local-llm-setup-guide-windows-11-wsl2-bionic-qwen-code.txt", "jsonld": "https://wpnews.pro/news/local-llm-setup-guide-windows-11-wsl2-bionic-qwen-code.jsonld"}}