Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

A developer behind SentioGate, an LLM routing service, described how the platform registers self-hosted inference endpoints such as vLLM, Ollama and LM Studio as "custom providers," letting apps call local and cloud models through a single OpenAI-compatible endpoint with unified request, streaming and telemetry formats. Models registered this way also become candidates for the router's "auto" mode, which selects a model per request based on capability, context window, cost and measured reliability, falling back to the next candidate before any bytes reach the client.

You have cloud keys in your app, and a GPU box in the corner running vLLM or Ollama. Here's how SentioGate https://sentiogate.app puts both behind one endpoint, with one request format, one stream format and one dashboard. A question I get a lot: "I use the cloud for the hard tasks, but I have my own box for the easy stuff. Can one router handle both, without a separate integration for each?" Yes, as long as your box speaks the OpenAI chat-completions API. vLLM, Ollama, LM Studio and most private inference appliances do. Most teams end up with something like this: if task difficulty 0.7: response = openai client.chat ... cloud else: response = ollama client.generate ... local That gives you two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry. Every new model means another branch, and every outage means a code change. A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost. In SentioGate, your own endpoint is a custom provider . It's a provider id , a base url , an api key , and the list of model IDs it serves. curl -X POST https://api.sentiogate.dev/v1/custom-providers \ -H "Authorization: Bearer $SENTIOGATE API KEY" \ -H "Content-Type: application/json" \ -d '{ "provider id": "my-vllm", "display name": "My vLLM", "base url": "https://llm.example.com/v1", "api key": "YOUR ENDPOINT KEY", "models": "llama3.1:8b", "qwen2.5:14b" }' A few things to get right: base url must be reachable from the internet. A URL like http://localhost:11434 or a LAN hostname won't work. Put the endpoint behind HTTPS. --api-key , and for Ollama a small reverse proxy that checks a bearer token does the job. api key . Once registered, you can call a model explicitly as provider id/model : python from ai auto router client import AutoRouterClient router = AutoRouterClient api key="sg live ..." resp = router.chat model="my-vllm/qwen2.5:14b", messages= {"role": "user", "content": "Summarise this ticket in one line: ..."} , print resp "choices" 0 "message" "content" Or from JavaScript: js import { AutoRouter } from '@hushamdev/ai-auto-router-sdk'; const router = new AutoRouter { apiKey: process.env.SENTIOGATE API KEY } ; const res = await router.chat { model: 'my-vllm/qwen2.5:14b', messages: { role: 'user', content: 'Summarise this ticket in one line: ...' } , } ; console.log res.choices 0 .message.content ; Your models also become candidates for "model": "auto" , right next to the cloud models your keys can reach. auto treats your models With auto , SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently. That last part is measured, not configured: if your box slows down or starts erroring, it drops down the list, and requests fall back to the next candidate before any bytes reach your client. You stay in control of the pool: my-vllm/... , or enable only your own models in the Vault so And you never have to guess what happened. Every response says which model was picked and why, in the router block, or in the X-SentioGate-Router header when streaming. Your models go through the same request path as the cloud ones: router.compression in the response shows what was saved. Streaming from your own model looks exactly like streaming from OpenAI: js for await const chunk of router.streamChat { model: 'my-vllm/qwen2.5:14b', messages: { role: 'user', content: 'Write a haiku about GPUs' } , } { process.stdout.write chunk.choices?. 0 ?.delta?.content || '' ; } Python: router.stream chat ... . The setup I'd recommend: You can mix all three in one app with one client, and pass cloud keys per request provider keys / providerKeys if you'd rather not store them. Keys passed that way are used for that request only. It's a simple mental model: explicit when you know what you want, auto when you want the router to decide, and one response format either way. Want everything, router included, on your own infrastructure? The source isn't public yet, so get in touch. ai-auto-router-client · @hushamdev/ai-auto-router-sdk Running a local box next to your cloud keys? Tell me in the comments what you serve on it. I'm curious which workloads people keep on their own hardware.