# Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

> Source: <https://dev.to/sentiogate-org/ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys-1eb4>
> Published: 2026-10-20 16:15:12+00:00

You have cloud keys in your app, and a GPU box in the corner running vLLM or Ollama. Here's how [SentioGate](https://sentiogate.app) puts both behind one endpoint, with one request format, one stream format and one dashboard.

A question I get a lot: "I use the cloud for the hard tasks, but I have my own box for the easy stuff. Can one router handle both, without a separate integration for each?"

Yes, as long as your box speaks the OpenAI chat-completions API. vLLM, Ollama, LM Studio and most private inference appliances do.

Most teams end up with something like this:

```
if task_difficulty > 0.7:
    response = openai_client.chat(...)      # cloud
else:
    response = ollama_client.generate(...)  # local
```

That gives you two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry. Every new model means another branch, and every outage means a code change.

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

In SentioGate, your own endpoint is a **custom provider**. It's a `provider_id`, a `base_url`, an `api_key`, and the list of model IDs it serves.

```
curl -X POST https://api.sentiogate.dev/v1/custom-providers \
  -H "Authorization: Bearer $SENTIOGATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider_id": "my-vllm",
    "display_name": "My vLLM",
    "base_url": "https://llm.example.com/v1",
    "api_key": "YOUR_ENDPOINT_KEY",
    "models": ["llama3.1:8b", "qwen2.5:14b"]
  }'
```

A few things to get right:

`base_url` must be reachable from the internet. A URL like `http://localhost:11434` or a LAN hostname won't work. Put the endpoint behind HTTPS.`--api-key`, and for Ollama a small reverse proxy that checks a bearer token does the job.`api_key`.
Once registered, you can call a model explicitly as `provider_id/model`:

``` python
from ai_auto_router_client import AutoRouterClient

router = AutoRouterClient(api_key="sg_live_...")

resp = router.chat(
    model="my-vllm/qwen2.5:14b",
    messages=[{"role": "user", "content": "Summarise this ticket in one line: ..."}],
)
print(resp["choices"][0]["message"]["content"])
```

Or from JavaScript:

``` js
import { AutoRouter } from '@hushamdev/ai-auto-router-sdk';

const router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY });
const res = await router.chat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Summarise this ticket in one line: ...' }],
});
console.log(res.choices[0].message.content);
```

Your models also become candidates for `"model": "auto"`, right next to the cloud models your keys can reach.

`auto` treats your models
With `auto`, SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently. That last part is measured, not configured: if your box slows down or starts erroring, it drops down the list, and requests fall back to the next candidate before any bytes reach your client.

You stay in control of the pool:

`my-vllm/...`), or enable only your own models in the Vault so And you never have to guess what happened. Every response says which model was picked and why, in the `router` block, or in the `X-SentioGate-Router` header when streaming.

Your models go through the same request path as the cloud ones:

`router.compression` in the response shows what was saved.
Streaming from your own model looks exactly like streaming from OpenAI:

``` js
for await (const chunk of router.streamChat({
  model: 'my-vllm/qwen2.5:14b',
  messages: [{ role: 'user', content: 'Write a haiku about GPUs' }],
})) {
  process.stdout.write(chunk.choices?.[0]?.delta?.content || '');
}
```

(Python: `router.stream_chat(...)`.)

The setup I'd recommend:

You can mix all three in one app with one client, and pass cloud keys per request (`provider_keys` / `providerKeys`) if you'd rather not store them. Keys passed that way are used for that request only.

It's a simple mental model: explicit when you know what you want, `auto` when you want the router to decide, and one response format either way.

Want everything, router included, on your own infrastructure? The source isn't public yet, so get in touch.

`ai-auto-router-client`) · `@hushamdev/ai-auto-router-sdk`)
Running a local box next to your cloud keys? Tell me in the comments what you serve on it. I'm curious which workloads people keep on their own hardware.