cd /news/large-language-models/local-llm-setup-guide-windows-11-wsl… · home › topics › large-language-models › article
[ARTICLE · art-152340] src=gist.github.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Local LLM Setup Guide — Windows 11 + WSL2 + Bionic + Qwen Code

A developer documented a troubleshooting guide for running local LLMs on a Windows 11 + WSL2 setup, pairing the Bionic (LM Studio fork) backend with the Qwen Code CLI and llama.cpp on CUDA 12. The guide identifies concrete pitfalls — WSL2 port 1234 collisions, localhost isolation between WSL2 and Windows, silent CPU fallback from misconfigured speculative decoding, and CUDA failures when running from an external drive — and prescribes fixes including mirrored networking, port 8000, a 14B Qwen2.5-Coder model with ~29 GPU-offloaded layers, and a 32768 context length to accommodate Qwen Code's ~20,000-token system prompt.

key takeaway

On an 8GB RTX 3070 laptop, running Qwen2.5-Coder-14B with ~29 GPU-offloaded layers, Q8_0 KV cache quantization, Flash Attention and a 32768 context length avoids the CPU fallback and request rejections caused by speculative decoding without a draft model and a 4096 context.

by read3 min views80 publishedOct 5, 2026

Here is the definitive, concise record of our troubleshooting journey. Save this as a reference guide (e.g., LOCAL_LLM_SETUP_GUIDE.md). It is structured so that any newcomer, AI instance, or developer can understand the exact architectural pitfalls of running local LLMs on a Windows + WSL2 + Bionic (LM Studio) setup.

Hardware Profile: ASUS Laptop, AMD Ryzen 9 5900HX, NVIDIA RTX 3070 Laptop (8GB VRAM), 64GB RAM, Windows 11, WSL2 (Ubuntu).

  • Backend: Bionic (LM Studio fork) running on Windows.

  • Frontend: Qwen Code CLI running inside WSL2.

  • Inference:llama.cpp with CUDA 12 runtime.

  • Cause: WSL2 dynamically reserves ports on Windows startup. Port1234 is frequently stolen by Windows services (svchost ,winnat ).

  • **Fix:**Never use port 1234. Change Bionic's API port to8000 . Ports in the 8000 range are ignored by WSL2's dynamic reservation.

  • Action: In Bionic:Local Model API ->Edit port ->8000 .

  • Cause:localhost inside WSL2 is not the same aslocalhost on Windows. WSL2 cannot see services bound to Windows127.0.0.1 .

  • Fix A (Recommended): Enable mirrored networking. CreateC:\Users\periman\.wslconfig and add: Then run

[wsl2]
networkingMode=mirrored

wsl --shutdown in PowerShell (Admin).

  • Fix B (Alternative): Bind Bionic to all interfaces (--bind 0.0.0.0 ) and use the Windows host IP (found in/etc/resolv.conf ) in your WSL scripts.

  • Cause: Speculative Decoding is enabled in the UI, but no draft model is selected. This causes Bionic to silently crash the model load, resulting in0% GPU usage and the model running entirely on CPU.

  • Fix: TurnSpeculative Decoding OFF for all models unless you have a compatible draft model configured.

  • Draft Model Bug: If you try to use a draft model, avoidQ8_0 quantized draft models on CUDA 12. They will crash witherror model: invalid vector subscript . UseQ4_K_M for drafts.

  • Cause 1: Running a 30B model on an 8GB GPU. Even with MoE CPU off (--n-cpu-moe ), the CPU prefill bottleneck makes it take 2.5 minutes to answer a simple prompt.

  • Cause 2: Running Bionic from anExternal HDD . Windows blocks CUDA initialization from external drives, forcing CPU-only inference.

  • Fix: 1. Move Bionic to the internal C: drive. 2. Use a 14B model (e.g.,Qwen2.5-Coder-14B-Instruct ). 3. Set GPU Offload to the maximum number of layers that fits in ~7.5GB VRAM (usually ~29 layers). 4. Set K Cache Quantization andV Cache Quantization toQ8_0 . 5. Enable Flash Attention andUnified KV Cache .

  • Move Bionic to the internal

  • Cause: Qwen Code uses an environment variableQWEN_HOME pointing to a specific directory (/home/periman/qwen-local-agent/qwen-home ). Editing~/.qwen/settings.json does nothing.

  • Fix: 1. Check printenv | grep QWEN . 2. Edit the actual settings.json inside theQWEN_HOME directory. 3. Ensure the "id" inmodelProvidersexactly matches the model ID exposed by Bionic's API (check withcurl -s http://localhost:8000/v1/models ). 4. Ensure "baseUrl" is set tohttp://localhost:8000/v1 .

  • Check

  • Cause: Qwen Code sends a massive ~20,000 token system prompt. If Bionic's model context is set to 4096, it rejects the request.

  • Fix: Set Bionic'sContext Length to32768 .

  • Model: Qwen2.5-Coder-14B-Instruct (or Q3_K_M for even more GPU offload).

  • GPU Offload: 29 layers (monitor Task ManagerGPU 1 , keep VRAM under 7.8GB).

  • Context Length: 32768.

  • K Cache Quantization: Q8_0.

  • V Cache Quantization: Q8_0.

  • Flash Attention: ON.

  • Unified KV Cache: ON.

  • Speculative Decoding: OFF.

  • Environment Variables:OPENAI_API_KEY="lm-studio" ,QWEN_HOME=/home/periman/qwen-local-agent/qwen-home .

  • Qwen Code Update: Runsudo npm install -g @qwen-code/qwen-code@latest . IfRipgrep EACCES error occurs, runsudo chmod +x /usr/lib/node_modules/@qwen-code/qwen-code/vendor/ripgrep/x64-linux/rg .

  1. Put Bionic on your C: drive.
  2. Use port 8000 .
  3. Use a 14B model, not 30B.
  4. Turn Speculative Decoding OFF.
  5. Find the real settings.json usingprintenv | grep QWEN .
  6. Match the model ID in the config exactly to Bionic's API output.
── more in #large-language-models 4 stories · sorted by recency
── more on @qwen code 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/local-llm-setup-guid…] indexed:0 read:3min 2026-10-05 · —