Here is the definitive, concise record of our troubleshooting journey. Save this as a reference guide (e.g., LOCAL_LLM_SETUP_GUIDE.md). It is structured so that any newcomer, AI instance, or developer can understand the exact architectural pitfalls of running local LLMs on a Windows + WSL2 + Bionic (LM Studio) setup.
Hardware Profile: ASUS Laptop, AMD Ryzen 9 5900HX, NVIDIA RTX 3070 Laptop (8GB VRAM), 64GB RAM, Windows 11, WSL2 (Ubuntu).
-
Backend: Bionic (LM Studio fork) running on Windows.
-
Frontend: Qwen Code CLI running inside WSL2.
-
Inference:
llama.cppwith CUDA 12 runtime. -
Cause: WSL2 dynamically reserves ports on Windows startup. Port
1234is frequently stolen by Windows services (svchost,winnat). -
**Fix:**Never use port 1234. Change Bionic's API port to
8000. Ports in the 8000 range are ignored by WSL2's dynamic reservation. -
Action: In Bionic:
Local Model API->Edit port->8000. -
Cause:
localhostinside WSL2 is not the same aslocalhoston Windows. WSL2 cannot see services bound to Windows127.0.0.1. -
Fix A (Recommended): Enable mirrored networking. Create
C:\Users\periman\.wslconfigand add: Then run
[wsl2]
networkingMode=mirrored
wsl --shutdown in PowerShell (Admin).
-
Fix B (Alternative): Bind Bionic to all interfaces (
--bind 0.0.0.0) and use the Windows host IP (found in/etc/resolv.conf) in your WSL scripts. -
Cause: Speculative Decoding is enabled in the UI, but no draft model is selected. This causes Bionic to silently crash the model load, resulting in0% GPU usage and the model running entirely on CPU.
-
Fix: TurnSpeculative Decoding OFF for all models unless you have a compatible draft model configured.
-
Draft Model Bug: If you try to use a draft model, avoid
Q8_0quantized draft models on CUDA 12. They will crash witherror model: invalid vector subscript. UseQ4_K_Mfor drafts. -
Cause 1: Running a 30B model on an 8GB GPU. Even with MoE CPU off (
--n-cpu-moe), the CPU prefill bottleneck makes it take 2.5 minutes to answer a simple prompt. -
Cause 2: Running Bionic from anExternal HDD . Windows blocks CUDA initialization from external drives, forcing CPU-only inference.
-
Fix: 1. Move Bionic to the internal
C:drive. 2. Use a 14B model (e.g.,Qwen2.5-Coder-14B-Instruct). 3. SetGPU Offloadto the maximum number of layers that fits in ~7.5GB VRAM (usually ~29 layers). 4. SetK Cache QuantizationandV Cache QuantizationtoQ8_0. 5. EnableFlash AttentionandUnified KV Cache. -
Move Bionic to the internal
-
Cause: Qwen Code uses an environment variable
QWEN_HOMEpointing to a specific directory (/home/periman/qwen-local-agent/qwen-home). Editing~/.qwen/settings.jsondoes nothing. -
Fix: 1. Check
printenv | grep QWEN. 2. Edit the actualsettings.jsoninside theQWEN_HOMEdirectory. 3. Ensure the"id"inmodelProvidersexactly matches the model ID exposed by Bionic's API (check withcurl -s http://localhost:8000/v1/models). 4. Ensure"baseUrl"is set tohttp://localhost:8000/v1. -
Check
-
Cause: Qwen Code sends a massive ~20,000 token system prompt. If Bionic's model context is set to 4096, it rejects the request.
-
Fix: Set Bionic's
Context Lengthto32768. -
Model: Qwen2.5-Coder-14B-Instruct (or Q3_K_M for even more GPU offload).
-
GPU Offload: 29 layers (monitor Task Manager
GPU 1, keep VRAM under 7.8GB). -
Context Length: 32768.
-
K Cache Quantization: Q8_0.
-
V Cache Quantization: Q8_0.
-
Flash Attention: ON.
-
Unified KV Cache: ON.
-
Speculative Decoding: OFF.
-
Environment Variables:
OPENAI_API_KEY="lm-studio",QWEN_HOME=/home/periman/qwen-local-agent/qwen-home. -
Qwen Code Update: Run
sudo npm install -g @qwen-code/qwen-code@latest. IfRipgrep EACCESerror occurs, runsudo chmod +x /usr/lib/node_modules/@qwen-code/qwen-code/vendor/ripgrep/x64-linux/rg.
- Put Bionic on your
C:drive. - Use port
8000. - Use a 14B model, not 30B.
- Turn
Speculative DecodingOFF. - Find the real
settings.jsonusingprintenv | grep QWEN. - Match the model ID in the config exactly to Bionic's API output.