# Run Qwen locally on Windows and use it remotely from any device > Source: > Published: 2026-08-17 21:44:33+00:00 Private local inference guide # Run Qwen locally on Windows and use it remotely from any device **Turn a high-RAM Windows PC into your personal AI server.** Use Qwen or any other local LLM from every laptop, desktop, and application you already own. Ollama runs the model, LiteLLM supplies an authenticated OpenAI-compatible endpoint, and Tailscale makes it available to your own devices without placing it on the public internet. **What we tested:** the complete path with a Qwen 3.5 9B GGUF model. Qwen 3.6 uses the same setup with a different model identifier and more memory. Qwen 3.8, just launched, requires a bit more configuration and optimization out of the box, covered here. ## The private inference architecture **Ollama**(or any other llama.cpp service) loads a local LLM model and listens for API calls only on the computer.** LiteLLM Proxy**presents one authenticated, OpenAI-compatible API and maps a stable alias to the installed model(s).** Tailscale Serve**terminates HTTPS and exposes LiteLLM only inside your tailnet. This guide does not enable Tailscale Funnel. Keeping both application servers on `127.0.0.1` prevents ordinary LAN or public access. Remote clients enter through the authenticated tailnet and still need the LiteLLM master key. ## Prerequisites - Windows 10 or 11 with Ollama installed and an LLM model already downloaded. - Tailscale installed and signed into the same tailnet as the client devices. - PowerShell running as Administrator for Tailscale Serve configuration. - Enough RAM or VRAM for the exact model and quantization you select. ## 1. Verify Ollama/Llama.cpp works before adding a router The fastest way to get started with older models is **ollama run.** ``` ollama run qwen3.5:9b ollama list Invoke-RestMethod http://127.0.0.1:11434/api/tags ``` Test the exact Qwen 3.5 model directly. This example uses the model from our experiment; substitute your actual model identifier when needed: ``` php $body = @{ model = "hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M" messages = @( @{ role = "user" content = "Reply with exactly: I'm working" } ) think = $false stream = $false } | ConvertTo-Json -Depth 6 Invoke-RestMethod ` -Uri "http://127.0.0.1:11434/api/chat" ` -Method Post ` -ContentType "application/json" ` -Body $body ``` The newer Qwen 3.8 27B model requires optimizations for faster performance. Community members have reported speedups exceeding 66 tok/s by going past default settings. We used llama.cpp instead of Ollama for the newer Qwen model, though Ollama still technically works. ``` php $SERVER = Get-ChildItem "C:\[Llama.cpp folder]" -Recurse -Filter llama-server.exe | Select-Object -First 1 $SERVER.FullName & $SERVER.FullName ` -m "C:\[path to model]\Qwen3.8-27B-UD-Q4_K_XL.gguf" ` --alias qwen38 ` --host 127.0.0.1 ` --port 8080 ` --ctx-size 65536 ` --n-gpu-layers auto ` --fit on ` --flash-attn on ` --cache-type-k q8_0 ` --cache-type-v q8_0 ` --parallel 1 ` --jinja ` --no-mmproj ` --perf ``` ## 2. Install a pinned LiteLLM Proxy ``` winget install --id=astral-sh.uv -e uv tool install --python 3.12 "litellm[proxy]==1.90.3" ``` The explicit version makes the tested environment reproducible. Review a later LiteLLM release before changing the pin. ## 3. Configure a stable model alias Create a neutral local directory and configuration file: ``` New-Item -ItemType Directory -Force C:\local-ai-router notepad C:\local-ai-router\config.yaml ``` For Ollama, use this configuration: ``` model_list: - model_name: local-qwen litellm_params: model: ollama_chat/qwen3.5:9b api_base: http://127.0.0.1:11434 general_settings: master_key: os.environ/LITELLM_MASTER_KEY ``` For the llama.cpp server configured above, use this configuration: ``` model_list: - model_name: local-qwen litellm_params: model: openai/qwen38 api_base: http://127.0.0.1:8080/v1 general_settings: master_key: os.environ/LITELLM_MASTER_KEY ``` Generate a master key beginning with "sk-", place it in the process environment, and start LiteLLM on loopback: ``` $env:LITELLM_MASTER_KEY = "sk-replace-with-a-long-random-secret" uv tool run litellm ` --config C:\local-ai-router\config.yaml ` --host 127.0.0.1 ` --port 4000 ``` ## 4. Test the authenticated local endpoint Open a second PowerShell window with the same master-key environment variable, then check health: ``` Invoke-RestMethod http://127.0.0.1:4000/health/liveliness ``` Send an OpenAI-compatible chat request through the model alias: ``` php $body = @{ model = "local-qwen" messages = @( @{ role = "user" content = "Reply with exactly: router works" } ) reasoning_effort = "none" stream = $false max_tokens = 40 } | ConvertTo-Json -Depth 6 Invoke-RestMethod ` -Uri "http://127.0.0.1:4000/v1/chat/completions" ` -Method Post ` -Headers @{ Authorization = "Bearer $env:LITELLM_MASTER_KEY" } ` -ContentType "application/json" ` -Body $body ``` ## 5. Get Tailscale and put authenticated inference on your private tailnet Download Tailscale and use the free plan at [https://tailscale.com/download/windows](https://tailscale.com/download/windows). Point Tailscale Serve at authenticated LiteLLM on port 4000. Do not point it at Ollama's unauthenticated port 11434. ``` & "C:\Program Files\Tailscale\tailscale.exe" serve reset & "C:\Program Files\Tailscale\tailscale.exe" ` serve --bg --https=443 http://127.0.0.1:4000 & "C:\Program Files\Tailscale\tailscale.exe" serve status & "C:\Program Files\Tailscale\tailscale.exe" ip -4 ``` Tailscale prints the private HTTPS hostname. A client must belong to the permitted tailnet and send the LiteLLM bearer token. Use `tailscale serve off` when the endpoint should no longer be available. The two internal services remain bound to loopback: `127.0.0.1:11434` : Ollama, available only on the inference box.`127.0.0.1:4000` : LiteLLM, available through private Tailscale HTTPS. ## 6. Test from another tailnet device On another authorized device, use the same OpenAI chat-completions body and point the request at the private hostname printed by Tailscale: ``` php Powershell version $headers = @{ Authorization = "Bearer sk-your-router-key" } Invoke-RestMethod ` -Uri "https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions" ` -Method Post ` -Headers $headers ` -ContentType "application/json" ` -Body $body Bash version cat > request.json <<'JSON' { "model": "local-qwen", "messages": [ { "role": "user", "content": "Reply with exactly: I'm working" } ], "stream": false, "max_tokens": 256 } JSON ROUTER_KEY="sk-your-router-key" URL="https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions" curl -sS "$URL" \ -H "Authorization: Bearer $ROUTER_KEY" \ -H "Content-Type: application/json" \ --data-binary @request.json ``` OpenAI-compatible applications can use that HTTPS origin as their base URL and the LiteLLM key as their API key. They do not need bespoke Ollama support. ## Diagnose one layer at a time ### Ollama succeeds; LiteLLM fails The model runtime works. Inspect the LiteLLM model identifier, YAML formatting, master key, and proxy logs. ### Ollama direct test fails The failure is below the router. Check whether Ollama is running, the model is installed, and the direct request uses the exact local model identifier. ### Both local tests succeed; remote fails The inference stack works. Inspect Tailscale Serve status, the private hostname, tailnet membership, and access-control policy. ## Qwen 3.5, 3.6, and 3.8 ### Qwen 3.5 The tested 9B Q4 model is the easiest starting point for a 32 GB Windows host. Keep the exact model identifier pinned in the LiteLLM configuration. ### Qwen 3.6 Ollama currently lists 27B and 35B variants. The same proxy and tailnet design applies, but download size and runtime memory are substantially larger. ### Qwen 3.8 Qwen 3.8 was just released with superior local coding abilities. We tested the model in this service arrangement with the Unsloth 4-bit quantization. While it was reasonable with memory use, it uses a LOT of thinking tokens and requires significant optimization to deploy. Users have been reporting every card has its own quirks to work out right now. ## Security checklist - Bind Ollama and LiteLLM to `127.0.0.1` , not every network interface. - Expose LiteLLM through Tailscale Serve; do not enable Funnel for this private endpoint. - Require a long LiteLLM master key and rotate it if it is copied into logs or shell history. - Use tailnet access controls to restrict which users and devices can reach the host. - Run the services under a dedicated non-administrator account for persistent operation. - Pin software versions and review updates before applying them to an inference host.