Local LLM Setup Guide — Windows 11 + WSL2 + Bionic + Qwen Code A developer documented a troubleshooting guide for running local LLMs on a Windows 11 + WSL2 setup, pairing the Bionic (LM Studio fork) backend with the Qwen Code CLI and llama.cpp on CUDA 12. The guide identifies concrete pitfalls — WSL2 port 1234 collisions, localhost isolation between WSL2 and Windows, silent CPU fallback from misconfigured speculative decoding, and CUDA failures when running from an external drive — and prescribes fixes including mirrored networking, port 8000, a 14B Qwen2.5-Coder model with ~29 GPU-offloaded layers, and a 32768 context length to accommodate Qwen Code's ~20,000-token system prompt. Here is the definitive, concise record of our troubleshooting journey. Save this as a reference guide e.g., LOCAL LLM SETUP GUIDE.md . It is structured so that any newcomer, AI instance, or developer can understand the exact architectural pitfalls of running local LLMs on a Windows + WSL2 + Bionic LM Studio setup. Hardware Profile: ASUS Laptop, AMD Ryzen 9 5900HX, NVIDIA RTX 3070 Laptop 8GB VRAM , 64GB RAM, Windows 11, WSL2 Ubuntu . - Backend: Bionic LM Studio fork running on Windows. - Frontend: Qwen Code CLI running inside WSL2. - Inference: llama.cpp with CUDA 12 runtime. - Cause: WSL2 dynamically reserves ports on Windows startup. Port 1234 is frequently stolen by Windows services svchost , winnat . - Fix: Never use port 1234. Change Bionic's API port to 8000 . Ports in the 8000 range are ignored by WSL2's dynamic reservation. - Action: In Bionic: Local Model API - Edit port - 8000 . - Cause: localhost inside WSL2 is not the same as localhost on Windows. WSL2 cannot see services bound to Windows 127.0.0.1 . - Fix A Recommended : Enable mirrored networking. Create C:\Users\periman\.wslconfig and add: Then run wsl2 networkingMode=mirrored wsl --shutdown in PowerShell Admin . - Fix B Alternative : Bind Bionic to all interfaces --bind 0.0.0.0 and use the Windows host IP found in /etc/resolv.conf in your WSL scripts. - Cause: Speculative Decoding is enabled in the UI, but no draft model is selected. This causes Bionic to silently crash the model load, resulting in 0% GPU usage and the model running entirely on CPU. - Fix: Turn Speculative Decoding OFF for all models unless you have a compatible draft model configured. - Draft Model Bug: If you try to use a draft model, avoid Q8 0 quantized draft models on CUDA 12. They will crash with error loading model: invalid vector subscript . Use Q4 K M for drafts. - Cause 1: Running a 30B model on an 8GB GPU. Even with MoE CPU offloading --n-cpu-moe , the CPU prefill bottleneck makes it take 2.5 minutes to answer a simple prompt. - Cause 2: Running Bionic from an External HDD . Windows blocks CUDA initialization from external drives, forcing CPU-only inference. - Fix: 1. Move Bionic to the internal C: drive. 2. Use a 14B model e.g., Qwen2.5-Coder-14B-Instruct . 3. Set GPU Offload to the maximum number of layers that fits in ~7.5GB VRAM usually ~29 layers . 4. Set K Cache Quantization and V Cache Quantization to Q8 0 . 5. Enable Flash Attention and Unified KV Cache . - Move Bionic to the internal - Cause: Qwen Code uses an environment variable QWEN HOME pointing to a specific directory /home/periman/qwen-local-agent/qwen-home . Editing ~/.qwen/settings.json does nothing. - Fix: 1. Check printenv | grep QWEN . 2. Edit the actual settings.json inside the QWEN HOME directory. 3. Ensure the "id" in modelProviders exactly matches the model ID exposed by Bionic's API check with curl -s http://localhost:8000/v1/models . 4. Ensure "baseUrl" is set to http://localhost:8000/v1 . - Check - Cause: Qwen Code sends a massive ~20,000 token system prompt. If Bionic's model context is set to 4096, it rejects the request. - Fix: Set Bionic's Context Length to 32768 . - Model: Qwen2.5-Coder-14B-Instruct or Q3 K M for even more GPU offload . - GPU Offload: 29 layers monitor Task Manager GPU 1 , keep VRAM under 7.8GB . - Context Length: 32768. - K Cache Quantization: Q8 0. - V Cache Quantization: Q8 0. - Flash Attention: ON. - Unified KV Cache: ON. - Speculative Decoding: OFF. - Environment Variables: OPENAI API KEY="lm-studio" , QWEN HOME=/home/periman/qwen-local-agent/qwen-home . - Qwen Code Update: Run sudo npm install -g @qwen-code/qwen-code@latest . If Ripgrep EACCES error occurs, run sudo chmod +x /usr/lib/node modules/@qwen-code/qwen-code/vendor/ripgrep/x64-linux/rg . 1. Put Bionic on your C: drive. 2. Use port 8000 . 3. Use a 14B model, not 30B. 4. Turn Speculative Decoding OFF. 5. Find the real settings.json using printenv | grep QWEN . 6. Match the model ID in the config exactly to Bionic's API output.