{"slug": "ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys", "title": "Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys", "summary": "A developer behind SentioGate, an LLM routing service, described how the platform registers self-hosted inference endpoints such as vLLM, Ollama and LM Studio as \"custom providers,\" letting apps call local and cloud models through a single OpenAI-compatible endpoint with unified request, streaming and telemetry formats. Models registered this way also become candidates for the router's \"auto\" mode, which selects a model per request based on capability, context window, cost and measured reliability, falling back to the next candidate before any bytes reach the client.", "body_md": "You have cloud keys in your app, and a GPU box in the corner running vLLM or Ollama. Here's how [SentioGate](https://sentiogate.app) puts both behind one endpoint, with one request format, one stream format and one dashboard.\n\nA question I get a lot: \"I use the cloud for the hard tasks, but I have my own box for the easy stuff. Can one router handle both, without a separate integration for each?\"\n\nYes, as long as your box speaks the OpenAI chat-completions API. vLLM, Ollama, LM Studio and most private inference appliances do.\n\nMost teams end up with something like this:\n\n```\nif task_difficulty > 0.7:\n    response = openai_client.chat(...)      # cloud\nelse:\n    response = ollama_client.generate(...)  # local\n```\n\nThat gives you two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry. Every new model means another branch, and every outage means a code change.\n\nA router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.\n\nIn SentioGate, your own endpoint is a **custom provider**. It's a `provider_id`, a `base_url`, an `api_key`, and the list of model IDs it serves.\n\n```\ncurl -X POST https://api.sentiogate.dev/v1/custom-providers \\\n  -H \"Authorization: Bearer $SENTIOGATE_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"provider_id\": \"my-vllm\",\n    \"display_name\": \"My vLLM\",\n    \"base_url\": \"https://llm.example.com/v1\",\n    \"api_key\": \"YOUR_ENDPOINT_KEY\",\n    \"models\": [\"llama3.1:8b\", \"qwen2.5:14b\"]\n  }'\n```\n\nA few things to get right:\n\n`base_url` must be reachable from the internet. A URL like `http://localhost:11434` or a LAN hostname won't work. Put the endpoint behind HTTPS.`--api-key`, and for Ollama a small reverse proxy that checks a bearer token does the job.`api_key`.\nOnce registered, you can call a model explicitly as `provider_id/model`:\n\n``` python\nfrom ai_auto_router_client import AutoRouterClient\n\nrouter = AutoRouterClient(api_key=\"sg_live_...\")\n\nresp = router.chat(\n    model=\"my-vllm/qwen2.5:14b\",\n    messages=[{\"role\": \"user\", \"content\": \"Summarise this ticket in one line: ...\"}],\n)\nprint(resp[\"choices\"][0][\"message\"][\"content\"])\n```\n\nOr from JavaScript:\n\n``` js\nimport { AutoRouter } from '@hushamdev/ai-auto-router-sdk';\n\nconst router = new AutoRouter({ apiKey: process.env.SENTIOGATE_API_KEY });\nconst res = await router.chat({\n  model: 'my-vllm/qwen2.5:14b',\n  messages: [{ role: 'user', content: 'Summarise this ticket in one line: ...' }],\n});\nconsole.log(res.choices[0].message.content);\n```\n\nYour models also become candidates for `\"model\": \"auto\"`, right next to the cloud models your keys can reach.\n\n`auto` treats your models\nWith `auto`, SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently. That last part is measured, not configured: if your box slows down or starts erroring, it drops down the list, and requests fall back to the next candidate before any bytes reach your client.\n\nYou stay in control of the pool:\n\n`my-vllm/...`), or enable only your own models in the Vault so And you never have to guess what happened. Every response says which model was picked and why, in the `router` block, or in the `X-SentioGate-Router` header when streaming.\n\nYour models go through the same request path as the cloud ones:\n\n`router.compression` in the response shows what was saved.\nStreaming from your own model looks exactly like streaming from OpenAI:\n\n``` js\nfor await (const chunk of router.streamChat({\n  model: 'my-vllm/qwen2.5:14b',\n  messages: [{ role: 'user', content: 'Write a haiku about GPUs' }],\n})) {\n  process.stdout.write(chunk.choices?.[0]?.delta?.content || '');\n}\n```\n\n(Python: `router.stream_chat(...)`.)\n\nThe setup I'd recommend:\n\nYou can mix all three in one app with one client, and pass cloud keys per request (`provider_keys` / `providerKeys`) if you'd rather not store them. Keys passed that way are used for that request only.\n\nIt's a simple mental model: explicit when you know what you want, `auto` when you want the router to decide, and one response format either way.\n\nWant everything, router included, on your own infrastructure? The source isn't public yet, so get in touch.\n\n`ai-auto-router-client`) · `@hushamdev/ai-auto-router-sdk`)\nRunning a local box next to your cloud keys? Tell me in the comments what you serve on it. I'm curious which workloads people keep on their own hardware.", "url": "https://wpnews.pro/news/ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys", "canonical_source": "https://dev.to/sentiogate-org/ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys-1eb4", "published_at": "2026-10-20 16:15:12+00:00", "updated_at": "2026-10-20 16:20:06.641165+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "ai-agents"], "entities": ["SentioGate", "Ollama", "vLLM", "LM Studio", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys", "markdown": "https://wpnews.pro/news/ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys.md", "text": "https://wpnews.pro/news/ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys.txt", "jsonld": "https://wpnews.pro/news/ollama-vllm-private-llms-routing-your-own-models-next-to-your-cloud-keys.jsonld"}}