{"slug": "run-qwen-locally-on-windows-and-use-it-remotely-from-any-device", "title": "Run Qwen locally on Windows and use it remotely from any device", "summary": "A new guide from an unnamed author details how to run Qwen locally on Windows and access it remotely from any device using Ollama, LiteLLM Proxy, and Tailscale. The setup was tested with a Qwen 3.5 9B GGUF model, and the guide also covers newer Qwen 3.6 and 3.8 models, with the latter requiring additional configuration. The architecture keeps servers on 127.0.0.1, uses LiteLLM for an authenticated OpenAI-compatible endpoint, and Tailscale Serve for secure remote access within a private tailnet.", "body_md": "Private local inference guide\n\n# Run Qwen locally on Windows and use it remotely from any device\n\n**Turn a high-RAM Windows PC into your personal AI server.** Use Qwen or any other local LLM from every laptop, desktop, and application you already own.\n\nOllama runs the model, LiteLLM supplies an authenticated OpenAI-compatible endpoint, and Tailscale makes it available to your own devices without placing it on the public internet.\n\n**What we tested:** the complete path with a Qwen 3.5 9B GGUF model. Qwen 3.6 uses the same setup with a different model identifier and more memory. Qwen 3.8, just launched, requires a bit more configuration and optimization out of the box, covered here.\n\n## The private inference architecture\n\n**Ollama**(or any other llama.cpp service) loads a local LLM model and listens for API calls only on the computer.** LiteLLM Proxy**presents one authenticated, OpenAI-compatible API and maps a stable alias to the installed model(s).** Tailscale Serve**terminates HTTPS and exposes LiteLLM only inside your tailnet. This guide does not enable Tailscale Funnel.\n\nKeeping both application servers on `127.0.0.1`\n\nprevents ordinary LAN or public access. Remote clients enter through the authenticated tailnet and still need the LiteLLM master key.\n\n## Prerequisites\n\n- Windows 10 or 11 with Ollama installed and an LLM model already downloaded.\n- Tailscale installed and signed into the same tailnet as the client devices.\n- PowerShell running as Administrator for Tailscale Serve configuration.\n- Enough RAM or VRAM for the exact model and quantization you select.\n\n## 1. Verify Ollama/Llama.cpp works before adding a router\n\nThe fastest way to get started with older models is **ollama run.**\n\n```\nollama run qwen3.5:9b\nollama list\n\nInvoke-RestMethod http://127.0.0.1:11434/api/tags\n```\n\nTest the exact Qwen 3.5 model directly. This example uses the model from our experiment; substitute your actual model identifier when needed:\n\n``` php\n$body = @{\n    model = \"hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M\"\n    messages = @(\n        @{\n            role = \"user\"\n            content = \"Reply with exactly: I'm working\"\n        }\n    )\n    think = $false\n    stream = $false\n} | ConvertTo-Json -Depth 6\n\nInvoke-RestMethod `\n    -Uri \"http://127.0.0.1:11434/api/chat\" `\n    -Method Post `\n    -ContentType \"application/json\" `\n    -Body $body\n```\n\nThe newer Qwen 3.8 27B model requires optimizations for faster performance. Community members have reported speedups exceeding 66 tok/s by going past default settings. We used llama.cpp instead of Ollama for the newer Qwen model, though Ollama still technically works.\n\n``` php\n$SERVER = Get-ChildItem \"C:\\[Llama.cpp folder]\" -Recurse -Filter llama-server.exe |\n  Select-Object -First 1\n\n$SERVER.FullName\n\n& $SERVER.FullName `\n  -m \"C:\\[path to model]\\Qwen3.8-27B-UD-Q4_K_XL.gguf\" `\n  --alias qwen38 `\n  --host 127.0.0.1 `\n  --port 8080 `\n  --ctx-size 65536 `\n  --n-gpu-layers auto `\n  --fit on `\n  --flash-attn on `\n  --cache-type-k q8_0 `\n  --cache-type-v q8_0 `\n  --parallel 1 `\n  --jinja `\n  --no-mmproj `\n  --perf\n```\n\n## 2. Install a pinned LiteLLM Proxy\n\n```\nwinget install --id=astral-sh.uv -e\nuv tool install --python 3.12 \"litellm[proxy]==1.90.3\"\n```\n\nThe explicit version makes the tested environment reproducible. Review a later LiteLLM release before changing the pin.\n\n## 3. Configure a stable model alias\n\nCreate a neutral local directory and configuration file:\n\n```\nNew-Item -ItemType Directory -Force C:\\local-ai-router\nnotepad C:\\local-ai-router\\config.yaml\n```\n\nFor Ollama, use this configuration:\n\n```\nmodel_list:\n  - model_name: local-qwen\n    litellm_params:\n      model: ollama_chat/qwen3.5:9b\n      api_base: http://127.0.0.1:11434\ngeneral_settings:\n  master_key: os.environ/LITELLM_MASTER_KEY\n```\n\nFor the llama.cpp server configured above, use this configuration:\n\n```\nmodel_list:\n  - model_name: local-qwen\n    litellm_params:\n      model: openai/qwen38\n      api_base: http://127.0.0.1:8080/v1\ngeneral_settings:\n  master_key: os.environ/LITELLM_MASTER_KEY\n```\n\nGenerate a master key beginning with \"sk-\", place it in the process environment, and start LiteLLM on loopback:\n\n```\n$env:LITELLM_MASTER_KEY = \"sk-replace-with-a-long-random-secret\"\n\nuv tool run litellm `\n  --config C:\\local-ai-router\\config.yaml `\n  --host 127.0.0.1 `\n  --port 4000\n```\n\n## 4. Test the authenticated local endpoint\n\nOpen a second PowerShell window with the same master-key environment variable, then check health:\n\n```\nInvoke-RestMethod http://127.0.0.1:4000/health/liveliness\n```\n\nSend an OpenAI-compatible chat request through the model alias:\n\n``` php\n$body = @{\n    model = \"local-qwen\"\n    messages = @(\n        @{\n            role = \"user\"\n            content = \"Reply with exactly: router works\"\n        }\n    )\n    reasoning_effort = \"none\"\n    stream = $false\n    max_tokens = 40\n} | ConvertTo-Json -Depth 6\n\nInvoke-RestMethod `\n    -Uri \"http://127.0.0.1:4000/v1/chat/completions\" `\n    -Method Post `\n    -Headers @{ Authorization = \"Bearer $env:LITELLM_MASTER_KEY\" } `\n    -ContentType \"application/json\" `\n    -Body $body\n```\n\n## 5. Get Tailscale and put authenticated inference on your private tailnet\n\nDownload Tailscale and use the free plan at [https://tailscale.com/download/windows](https://tailscale.com/download/windows). Point Tailscale Serve at authenticated LiteLLM on port 4000. Do not point it at Ollama's unauthenticated port 11434.\n\n```\n& \"C:\\Program Files\\Tailscale\\tailscale.exe\" serve reset\n\n& \"C:\\Program Files\\Tailscale\\tailscale.exe\" `\n    serve --bg --https=443 http://127.0.0.1:4000\n\n& \"C:\\Program Files\\Tailscale\\tailscale.exe\" serve status\n& \"C:\\Program Files\\Tailscale\\tailscale.exe\" ip -4\n```\n\nTailscale prints the private HTTPS hostname. A client must belong to the permitted tailnet and send the LiteLLM bearer token. Use `tailscale serve off`\n\nwhen the endpoint should no longer be available.\n\nThe two internal services remain bound to loopback:\n\n`127.0.0.1:11434`\n\n: Ollama, available only on the inference box.`127.0.0.1:4000`\n\n: LiteLLM, available through private Tailscale HTTPS.\n\n## 6. Test from another tailnet device\n\nOn another authorized device, use the same OpenAI chat-completions body and point the request at the private hostname printed by Tailscale:\n\n``` php\nPowershell version\n$headers = @{\n    Authorization = \"Bearer sk-your-router-key\"\n}\n\nInvoke-RestMethod `\n    -Uri \"https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions\" `\n    -Method Post `\n    -Headers $headers `\n    -ContentType \"application/json\" `\n    -Body $body\nBash version\ncat > request.json <<'JSON'\n{\n  \"model\": \"local-qwen\",\n  \"messages\": [\n    {\n      \"role\": \"user\",\n      \"content\": \"Reply with exactly: I'm working\"\n    }\n  ],\n  \"stream\": false,\n  \"max_tokens\": 256\n}\nJSON\n\nROUTER_KEY=\"sk-your-router-key\"\nURL=\"https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions\"\n\ncurl -sS \"$URL\" \\\n  -H \"Authorization: Bearer $ROUTER_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  --data-binary @request.json\n```\n\nOpenAI-compatible applications can use that HTTPS origin as their base URL and the LiteLLM key as their API key. They do not need bespoke Ollama support.\n\n## Diagnose one layer at a time\n\n### Ollama succeeds; LiteLLM fails\n\nThe model runtime works. Inspect the LiteLLM model identifier, YAML formatting, master key, and proxy logs.\n\n### Ollama direct test fails\n\nThe failure is below the router. Check whether Ollama is running, the model is installed, and the direct request uses the exact local model identifier.\n\n### Both local tests succeed; remote fails\n\nThe inference stack works. Inspect Tailscale Serve status, the private hostname, tailnet membership, and access-control policy.\n\n## Qwen 3.5, 3.6, and 3.8\n\n### Qwen 3.5\n\nThe tested 9B Q4 model is the easiest starting point for a 32 GB Windows host. Keep the exact model identifier pinned in the LiteLLM configuration.\n\n### Qwen 3.6\n\nOllama currently lists 27B and 35B variants. The same proxy and tailnet design applies, but download size and runtime memory are substantially larger.\n\n### Qwen 3.8\n\nQwen 3.8 was just released with superior local coding abilities. We tested the model in this service arrangement with the Unsloth 4-bit quantization. While it was reasonable with memory use, it uses a LOT of thinking tokens and requires significant optimization to deploy. Users have been reporting every card has its own quirks to work out right now.\n\n## Security checklist\n\n- Bind Ollama and LiteLLM to\n`127.0.0.1`\n\n, not every network interface. - Expose LiteLLM through Tailscale Serve; do not enable Funnel for this private endpoint.\n- Require a long LiteLLM master key and rotate it if it is copied into logs or shell history.\n- Use tailnet access controls to restrict which users and devices can reach the host.\n- Run the services under a dedicated non-administrator account for persistent operation.\n- Pin software versions and review updates before applying them to an inference host.", "url": "https://wpnews.pro/news/run-qwen-locally-on-windows-and-use-it-remotely-from-any-device", "canonical_source": "https://www.hiramdeals.com/blog/host-qwen-privately-windows", "published_at": "2026-08-17 21:44:33+00:00", "updated_at": "2026-08-17 22:11:25.633160+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Ollama", "LiteLLM", "Tailscale", "Qwen", "llama.cpp", "Windows"], "alternates": {"html": "https://wpnews.pro/news/run-qwen-locally-on-windows-and-use-it-remotely-from-any-device", "markdown": "https://wpnews.pro/news/run-qwen-locally-on-windows-and-use-it-remotely-from-any-device.md", "text": "https://wpnews.pro/news/run-qwen-locally-on-windows-and-use-it-remotely-from-any-device.txt", "jsonld": "https://wpnews.pro/news/run-qwen-locally-on-windows-and-use-it-remotely-from-any-device.jsonld"}}