{"slug": "free-llm-balancer-combines-multiple-local-inference-machines-with-cloud-fallback", "title": "Free LLM balancer combines multiple local inference machines with cloud fallback", "summary": "LLMrPro, a new self-hosted LLM router, pools users' own machines as inference workers with automatic per-tier cloud fallback, exposing a single OpenAI-compatible API. The open-source project, released in 2026, combines local inference engines like Ollama and vLLM with cloud providers including OpenAI and Anthropic, offering live pool management and security features such as per-source rate limiting and encrypted provider keys.", "body_md": "Your machines run the models. The cloud only fills the gaps.Call every model through one OpenAI format — self-hosted, and yours to own.\n\nLLMrPro is a self-hosted LLM router. It exposes **one OpenAI-compatible API** and serves each\nrequest from a pool of machines **you** own — desktops, workstations, servers — running local\ninference engines. When no local worker can serve a request, it transparently falls back to the\ncloud provider you configured for that tier. One endpoint, one request format, your hardware\nfirst, your keys, your data.\n\n**One OpenAI-compatible API for everything.** Point any OpenAI client or SDK at the balancer URL —`POST /{frontier,mini,nano}/v1/chat/completions`\n\nwith a standard Chat Completions body. Swap models and providers behind it without changing a line of caller code.**Pools your own machines as inference workers.** A lightweight desktop agent runs on each machine, connects**outbound** over a websocket (no inbound ports, no firewall changes), and pulls inference jobs. Each worker runs a full model on a local engine — LM Studio, Ollama, vLLM, llama.cpp, or the bundled MLX engine on macOS.**Automatic per-tier cloud fallback.** Three tiers —`frontier`\n\n,`mini`\n\n,`nano`\n\n— each with its own cloud backend (OpenAI, Anthropic, Google, Azure OpenAI). If the local pool can't serve a tier, the request falls back to that tier's cloud provider, with configurable retry/backoff.**Single-tenant and self-hosted.** One balancer per organization. Your keys, your data, your infrastructure — nothing leaves your control unless a request falls back to a cloud provider you configured.**Live pool management.** A built-in admin UI — device pairing, agent revocation, tier config, dispatch log, and health — backed by real-time pub/sub.**Security built in.** Per-source rate limiting, an egress guard (SSRF protection), provider keys encrypted at rest, signed agent tokens with expiry, and a hardened install/deploy path.\n\n```\n   OpenAI-compatible          ┌───────────────────────────────┐\n   client / app  ───────────▶ │            LLMrPro             │\n   (Bearer key)               │   balancer  (router + admin)   │\n                              │  tier: frontier / mini / nano  │\n                              └───────┬────────────────┬───────┘\n                        websocket     │ dispatch        │ fallback\n                        (job pull)    ▼ (local first)   ▼ (per tier)\n                       ┌───────────────────────┐   ┌───────────────────┐\n                       │  desktop agents        │   │  cloud providers  │\n                       │  (your machines)       │   │  OpenAI/Anthropic │\n                       │  MLX / Ollama / vLLM   │   │  / Google / Azure │\n                       └───────────────────────┘   └───────────────────┘\n```\n\n- A client sends a standard OpenAI Chat Completions request to a tier endpoint with the bearer key.\n- The balancer picks an eligible paired agent and dispatches the job; the agent runs it on its local engine and streams the result back.\n- If no agent can serve the tier, the balancer calls that tier's configured cloud provider and\nreturns the result in the same OpenAI format.\n`X-Balancer-*`\n\nresponse headers report who served the request and why.\n\nPlenty of projects touch part of this; as of 2026 no single open-source project combines all of it.\n\n| Compared to | What they do | What LLMrPro adds |\n|---|---|---|\nDistributed inference (exo, Petals) |\nSplit one model across devices via P2P / tensor parallelism | A job-dispatch worker pool (a full model per worker) plus per-tier cloud fallback, single-tenant |\nLLM gateways (LiteLLM, Bifrost) |\nRoute and fall back across static, pre-configured endpoints | A live pool of your own desktops that pair and pull jobs — no static config — plus a signed end-user agent |\nGPU-fleet managers (GPUStack, Kalavai) |\nWorker/coordinator scheduling for owned GPUs | Consumer desktops over the internet (incl. macOS/MLX), a one-click signed agent, and tiered cloud fallback |\nBalancer + agent tools (Paddler, HiveCore) |\nA balancer with self-registering pull workers | Per-tier cloud fallback and a signed desktop app built for non-technical users |\n\nThe distinctive combination — **federate a fleet of end-user desktops via a signed agent +\ntiered cloud fallback + single-tenant self-hosting, behind one OpenAI-compatible API** — is what\nLLMrPro is for.\n\n```\nLLMrPro/\n├── balancer/     # the router: Meteor server + MongoDB + React admin UI\n├── agent/        # the desktop worker: Electron + React app\n├── docs/         # operator documentation (deploy, API, agent install, troubleshooting)\n├── infra/        # provisioning + deploy scripts (Debian VM, reverse proxy, systemd)\n├── DESIGN.md     # full architecture, wire protocol, routing algorithm\n└── SECURITY.md   # security model + how to report a vulnerability\n```\n\nLLMrPro has two installable pieces: the **balancer** (you run one, server-side) and the\n**agent** (installed on each worker machine).\n\n**Requirements:** Node.js 22+, MongoDB 6+ (a replica set is required for Meteor 3 change-stream\nreactivity), Meteor 3.4+, and — in production — a TLS-terminating reverse proxy\n(nginx / Caddy / Traefik).\n\n**Local / development**\n\n```\ngit clone https://github.com/gysho/LLMrPro.git\ncd LLMrPro/balancer\nmeteor npm install\ncp .env.production.example .env.production   # then fill in the secrets (see below)\nnpm run dev                                  # inject-secrets + meteor run on :3500\n```\n\nOpen the admin UI at `http://localhost:3500/admin`\n\n.\n\n**Production — provision a fresh host** (Debian 12; installs Node, a MongoDB replica set, nginx,\na systemd unit, and ufw):\n\n```\nsudo bash infra/azure/setup-vm.sh balancer.example.com\n```\n\n**Production — deploy a release** (build → rsync → atomic symlink swap → restart):\n\n```\ncd balancer\n./scripts/deploy.sh https://balancer.example.com deploy@your-host\ncurl -i https://balancer.example.com/healthz   # expect: 200 ok\n```\n\nFull walkthrough: [docs/operator-runbook.md](/Gysho/LLMrPro/blob/main/docs/operator-runbook.md).\n\n**Secrets** are never committed. `settings.json`\n\nholds `${VAR}`\n\nplaceholders; real values live in\n`.env.production`\n\nand are injected into `settings.runtime.json`\n\nat boot by `inject-secrets.sh`\n\n.\nYou provide: a platform bearer key (stored only as a SHA-256 hash), an agent-token HMAC signing\nkey, the bootstrap admin password, and one cloud API key per tier. See\n[docs/operator-runbook.md §2](/Gysho/LLMrPro/blob/main/docs/operator-runbook.md).\n\nEach worker machine runs the agent, which connects outbound to the balancer (no inbound ports) and forwards jobs to a local inference engine.\n\n**End users — one-line install** (serves the right artifact for the OS/arch and verifies its\nSHA-256):\n\n```\ncurl -fsSL https://balancer.example.com/install.sh | sh\n```\n\nInspect it first with `curl -fsSL https://balancer.example.com/install.sh.txt`\n\n. On first launch\nthe agent opens a **Pairing** screen — paste the balancer URL and a pairing code (an admin\ngenerates these in the balancer UI), and the machine joins the pool.\n\n**Per platform**\n\n**macOS**(12+, Apple Silicon or Intel):`.dmg`\n\n, or the curl installer. Ships a bundled MLX inference engine; you can also point it at LM Studio / Ollama.**Linux**(x86_64):`.AppImage`\n\n(primary),`.deb`\n\n, or`.rpm`\n\n. Requires a local engine (LM Studio / Ollama / vLLM / llama.cpp). Manual install:`sudo apt install ./<agent>.deb`\n\nor`sudo rpm -i <agent>.rpm`\n\n.- Manual downloads are listed at\n`https://balancer.example.com/agent/downloads`\n\n.\n\n**Build the agent from source**\n\n```\ncd agent\nnpm install\nnpm run build:linux      # .AppImage / .deb / .rpm   (on a Linux host)\nnpm run build:mac        # .dmg + bundled MLX engine (on a macOS host)\n```\n\nThe desktop agent is distributed **unsigned** by design — see [SECURITY.md](/Gysho/LLMrPro/blob/main/SECURITY.md).\n\n```\ncurl https://balancer.example.com/frontier/v1/chat/completions \\\n  -H \"Authorization: Bearer <platform-key>\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n        \"model\": \"gpt-5\",\n        \"messages\": [{ \"role\": \"user\", \"content\": \"Hello\" }]\n      }'\n```\n\n**Endpoints:**`POST /{frontier,mini,nano}/v1/chat/completions`\n\n— the tier is the URL path.**Body:** standard OpenAI**Chat Completions**. Streaming, tools, and structured outputs pass through.** Auth:**`Authorization: Bearer <platform-key>`\n\n(required on every tier).- The response is OpenAI-formatted whether served by a local agent or a cloud fallback. Full\ncontract:\n[docs/api-contract.md](/Gysho/LLMrPro/blob/main/docs/api-contract.md).\n\nAll server config lives in `balancer/settings.json`\n\n(template: `balancer/settings-example.json`\n\n).\nKey blocks:\n\n`publicHostname`\n\n/`urlSuffix`\n\n— the public origin (and reverse-proxy path prefix, if any).`tiers.{frontier,mini,nano}.cloudFallback`\n\n— per-tier cloud backend:`provider`\n\n(openai / anthropic / google / azure),`apiSurface`\n\n,`endpoint`\n\n,`apiKey`\n\n,`deployment`\n\n,`capabilities`\n\n,`parameterMap`\n\n.`rateLimits`\n\n—`completionsPerSecond`\n\n,`pairAttemptsPerMinutePerIp`\n\n.`trustProxy`\n\n— set`true`\n\nbehind a trusted reverse proxy to get per-client rate limiting.`egressAllowlist`\n\n— optional hostname allowlist for outbound cloud calls (SSRF hardening).`cloudRetry`\n\n,`healthCheck`\n\n,`pairingCodeTtlMs`\n\n,`dispatchTimeoutMs`\n\n,`maxInflightPerAgent`\n\n.\n\nSecrets in `settings.json`\n\nare `${VAR}`\n\nplaceholders resolved from the environment at boot —\nnever commit real values.\n\n[Operator runbook](/Gysho/LLMrPro/blob/main/docs/operator-runbook.md)— deploy, configure tiers, rotate keys, monitor, update, roll back[Architecture](/Gysho/LLMrPro/blob/main/docs/architecture.md)·[full design](/Gysho/LLMrPro/blob/main/DESIGN.md)[API contract](/Gysho/LLMrPro/blob/main/docs/api-contract.md)- Agent install:\n[macOS](/Gysho/LLMrPro/blob/main/docs/agent-install-guide-mac.md)·[Linux](/Gysho/LLMrPro/blob/main/docs/agent-install-guide-linux.md) [Troubleshooting](/Gysho/LLMrPro/blob/main/docs/troubleshooting.md)·[FAQ](/Gysho/LLMrPro/blob/main/docs/faq.md)\n\nThe security model, the unsigned-agent distribution tradeoff, and how to report a vulnerability\nare in [SECURITY.md](/Gysho/LLMrPro/blob/main/SECURITY.md). Provider keys are encrypted at rest, agent tokens are signed\nwith expiry, and the balancer ships with rate limiting and SSRF egress protection.\n\nLLMrPro is released under the [MIT License](/Gysho/LLMrPro/blob/main/LICENSE). © 2026 Gysho.", "url": "https://wpnews.pro/news/free-llm-balancer-combines-multiple-local-inference-machines-with-cloud-fallback", "canonical_source": "https://github.com/Gysho/LLMrPro", "published_at": "2026-07-20 21:24:57+00:00", "updated_at": "2026-07-20 21:53:12.371505+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["LLMrPro", "OpenAI", "Anthropic", "Google", "Azure OpenAI", "Ollama", "vLLM", "LM Studio"], "alternates": {"html": "https://wpnews.pro/news/free-llm-balancer-combines-multiple-local-inference-machines-with-cloud-fallback", "markdown": "https://wpnews.pro/news/free-llm-balancer-combines-multiple-local-inference-machines-with-cloud-fallback.md", "text": "https://wpnews.pro/news/free-llm-balancer-combines-multiple-local-inference-machines-with-cloud-fallback.txt", "jsonld": "https://wpnews.pro/news/free-llm-balancer-combines-multiple-local-inference-machines-with-cloud-fallback.jsonld"}}