Run Qwen locally on Windows and use it remotely from any device

wpnews.pro

Private local inference guide

Turn a high-RAM Windows PC into your personal AI server. Use Qwen or any other local LLM from every laptop, desktop, and application you already own.

Ollama runs the model, LiteLLM supplies an authenticated OpenAI-compatible endpoint, and Tailscale makes it available to your own devices without placing it on the public internet.

What we tested: the complete path with a Qwen 3.5 9B GGUF model. Qwen 3.6 uses the same setup with a different model identifier and more memory. Qwen 3.8, just launched, requires a bit more configuration and optimization out of the box, covered here.

The private inference architecture #

Ollama(or any other llama.cpp service) loads a local LLM model and listens for API calls only on the computer.** LiteLLM Proxypresents one authenticated, OpenAI-compatible API and maps a stable alias to the installed model(s). Tailscale Serve**terminates HTTPS and exposes LiteLLM only inside your tailnet. This guide does not enable Tailscale Funnel.

Keeping both application servers on 127.0.0.1

prevents ordinary LAN or public access. Remote clients enter through the authenticated tailnet and still need the LiteLLM master key.

Prerequisites #

Windows 10 or 11 with Ollama installed and an LLM model already downloaded.
Tailscale installed and signed into the same tailnet as the client devices.
PowerShell running as Administrator for Tailscale Serve configuration.
Enough RAM or VRAM for the exact model and quantization you select.

1. Verify Ollama/Llama.cpp works before adding a router #

The fastest way to get started with older models is ollama run.

ollama run qwen3.5:9b
ollama list

Invoke-RestMethod http://127.0.0.1:11434/api/tags

Test the exact Qwen 3.5 model directly. This example uses the model from our experiment; substitute your actual model identifier when needed:

$body = @{
    model = "hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M"
    messages = @(
        @{
            role = "user"
            content = "Reply with exactly: I'm working"
        }
    )
    think = $false
    stream = $false
} | ConvertTo-Json -Depth 6

Invoke-RestMethod `
    -Uri "http://127.0.0.1:11434/api/chat" `
    -Method Post `
    -ContentType "application/json" `
    -Body $body

The newer Qwen 3.8 27B model requires optimizations for faster performance. Community members have reported speedups exceeding 66 tok/s by going past default settings. We used llama.cpp instead of Ollama for the newer Qwen model, though Ollama still technically works.

$SERVER = Get-ChildItem "C:\[Llama.cpp folder]" -Recurse -Filter llama-server.exe |
  Select-Object -First 1

$SERVER.FullName

& $SERVER.FullName `
  -m "C:\[path to model]\Qwen3.8-27B-UD-Q4_K_XL.gguf" `
  --alias qwen38 `
  --host 127.0.0.1 `
  --port 8080 `
  --ctx-size 65536 `
  --n-gpu-layers auto `
  --fit on `
  --flash-attn on `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --parallel 1 `
  --jinja `
  --no-mmproj `
  --perf

2. Install a pinned LiteLLM Proxy #

winget install --id=astral-sh.uv -e
uv tool install --python 3.12 "litellm[proxy]==1.90.3"

The explicit version makes the tested environment reproducible. Review a later LiteLLM release before changing the pin.

3. Configure a stable model alias #

Create a neutral local directory and configuration file:

New-Item -ItemType Directory -Force C:\local-ai-router
notepad C:\local-ai-router\config.yaml

For Ollama, use this configuration:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3.5:9b
      api_base: http://127.0.0.1:11434
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

For the llama.cpp server configured above, use this configuration:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: openai/qwen38
      api_base: http://127.0.0.1:8080/v1
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Generate a master key beginning with "sk-", place it in the process environment, and start LiteLLM on loopback:

$env:LITELLM_MASTER_KEY = "sk-replace-with-a-long-random-secret"

uv tool run litellm `
  --config C:\local-ai-router\config.yaml `
  --host 127.0.0.1 `
  --port 4000

4. Test the authenticated local endpoint #

Open a second PowerShell window with the same master-key environment variable, then check health:

Invoke-RestMethod http://127.0.0.1:4000/health/liveliness

Send an OpenAI-compatible chat request through the model alias:

$body = @{
    model = "local-qwen"
    messages = @(
        @{
            role = "user"
            content = "Reply with exactly: router works"
        }
    )
    reasoning_effort = "none"
    stream = $false
    max_tokens = 40
} | ConvertTo-Json -Depth 6

Invoke-RestMethod `
    -Uri "http://127.0.0.1:4000/v1/chat/completions" `
    -Method Post `
    -Headers @{ Authorization = "Bearer $env:LITELLM_MASTER_KEY" } `
    -ContentType "application/json" `
    -Body $body

5. Get Tailscale and put authenticated inference on your private tailnet #

Download Tailscale and use the free plan at https://tailscale.com/download/windows. Point Tailscale Serve at authenticated LiteLLM on port 4000. Do not point it at Ollama's unauthenticated port 11434.

& "C:\Program Files\Tailscale\tailscale.exe" serve reset

& "C:\Program Files\Tailscale\tailscale.exe" `
    serve --bg --https=443 http://127.0.0.1:4000

& "C:\Program Files\Tailscale\tailscale.exe" serve status
& "C:\Program Files\Tailscale\tailscale.exe" ip -4

Tailscale prints the private HTTPS hostname. A client must belong to the permitted tailnet and send the LiteLLM bearer token. Use tailscale serve off

when the endpoint should no longer be available.

The two internal services remain bound to loopback:

127.0.0.1:11434

: Ollama, available only on the inference box.127.0.0.1:4000

: LiteLLM, available through private Tailscale HTTPS.

6. Test from another tailnet device #

On another authorized device, use the same OpenAI chat-completions body and point the request at the private hostname printed by Tailscale:

Powershell version
$headers = @{
    Authorization = "Bearer sk-your-router-key"
}

Invoke-RestMethod `
    -Uri "https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions" `
    -Method Post `
    -Headers $headers `
    -ContentType "application/json" `
    -Body $body
Bash version
cat > request.json <<'JSON'
{
  "model": "local-qwen",
  "messages": [
    {
      "role": "user",
      "content": "Reply with exactly: I'm working"
    }
  ],
  "stream": false,
  "max_tokens": 256
}
JSON

ROUTER_KEY="sk-your-router-key"
URL="https://YOUR-INFERENCE-BOX.YOUR-TAILNET.ts.net/v1/chat/completions"

curl -sS "$URL" \
  -H "Authorization: Bearer $ROUTER_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

OpenAI-compatible applications can use that HTTPS origin as their base URL and the LiteLLM key as their API key. They do not need bespoke Ollama support.

Diagnose one layer at a time #

Ollama succeeds; LiteLLM fails

The model runtime works. Inspect the LiteLLM model identifier, YAML formatting, master key, and proxy logs.

Ollama direct test fails

The failure is below the router. Check whether Ollama is running, the model is installed, and the direct request uses the exact local model identifier.

Both local tests succeed; remote fails

The inference stack works. Inspect Tailscale Serve status, the private hostname, tailnet membership, and access-control policy.

Qwen 3.5, 3.6, and 3.8 #

Qwen 3.5

The tested 9B Q4 model is the easiest starting point for a 32 GB Windows host. Keep the exact model identifier pinned in the LiteLLM configuration.

Qwen 3.6

Ollama currently lists 27B and 35B variants. The same proxy and tailnet design applies, but download size and runtime memory are substantially larger.

Qwen 3.8

Qwen 3.8 was just released with superior local coding abilities. We tested the model in this service arrangement with the Unsloth 4-bit quantization. While it was reasonable with memory use, it uses a LOT of thinking tokens and requires significant optimization to deploy. Users have been reporting every card has its own quirks to work out right now.

Security checklist #

Bind Ollama and LiteLLM to 127.0.0.1

, not every network interface. - Expose LiteLLM through Tailscale Serve; do not enable Funnel for this private endpoint.

Require a long LiteLLM master key and rotate it if it is copied into logs or shell history.
Use tailnet access controls to restrict which users and devices can reach the host.
Run the services under a dedicated non-administrator account for persistent operation.
Pin software versions and review updates before applying them to an inference host.

source & further reading

hiramdeals.com — original article