Free LLM balancer combines multiple local inference machines with cloud fallback LLMrPro, a new self-hosted LLM router, pools users' own machines as inference workers with automatic per-tier cloud fallback, exposing a single OpenAI-compatible API. The open-source project, released in 2026, combines local inference engines like Ollama and vLLM with cloud providers including OpenAI and Anthropic, offering live pool management and security features such as per-source rate limiting and encrypted provider keys. Your machines run the models. The cloud only fills the gaps.Call every model through one OpenAI format — self-hosted, and yours to own. LLMrPro is a self-hosted LLM router. It exposes one OpenAI-compatible API and serves each request from a pool of machines you own — desktops, workstations, servers — running local inference engines. When no local worker can serve a request, it transparently falls back to the cloud provider you configured for that tier. One endpoint, one request format, your hardware first, your keys, your data. One OpenAI-compatible API for everything. Point any OpenAI client or SDK at the balancer URL — POST /{frontier,mini,nano}/v1/chat/completions with a standard Chat Completions body. Swap models and providers behind it without changing a line of caller code. Pools your own machines as inference workers. A lightweight desktop agent runs on each machine, connects outbound over a websocket no inbound ports, no firewall changes , and pulls inference jobs. Each worker runs a full model on a local engine — LM Studio, Ollama, vLLM, llama.cpp, or the bundled MLX engine on macOS. Automatic per-tier cloud fallback. Three tiers — frontier , mini , nano — each with its own cloud backend OpenAI, Anthropic, Google, Azure OpenAI . If the local pool can't serve a tier, the request falls back to that tier's cloud provider, with configurable retry/backoff. Single-tenant and self-hosted. One balancer per organization. Your keys, your data, your infrastructure — nothing leaves your control unless a request falls back to a cloud provider you configured. Live pool management. A built-in admin UI — device pairing, agent revocation, tier config, dispatch log, and health — backed by real-time pub/sub. Security built in. Per-source rate limiting, an egress guard SSRF protection , provider keys encrypted at rest, signed agent tokens with expiry, and a hardened install/deploy path. OpenAI-compatible ┌───────────────────────────────┐ client / app ───────────▶ │ LLMrPro │ Bearer key │ balancer router + admin │ │ tier: frontier / mini / nano │ └───────┬────────────────┬───────┘ websocket │ dispatch │ fallback job pull ▼ local first ▼ per tier ┌───────────────────────┐ ┌───────────────────┐ │ desktop agents │ │ cloud providers │ │ your machines │ │ OpenAI/Anthropic │ │ MLX / Ollama / vLLM │ │ / Google / Azure │ └───────────────────────┘ └───────────────────┘ - A client sends a standard OpenAI Chat Completions request to a tier endpoint with the bearer key. - The balancer picks an eligible paired agent and dispatches the job; the agent runs it on its local engine and streams the result back. - If no agent can serve the tier, the balancer calls that tier's configured cloud provider and returns the result in the same OpenAI format. X-Balancer- response headers report who served the request and why. Plenty of projects touch part of this; as of 2026 no single open-source project combines all of it. | Compared to | What they do | What LLMrPro adds | |---|---|---| Distributed inference exo, Petals | Split one model across devices via P2P / tensor parallelism | A job-dispatch worker pool a full model per worker plus per-tier cloud fallback, single-tenant | LLM gateways LiteLLM, Bifrost | Route and fall back across static, pre-configured endpoints | A live pool of your own desktops that pair and pull jobs — no static config — plus a signed end-user agent | GPU-fleet managers GPUStack, Kalavai | Worker/coordinator scheduling for owned GPUs | Consumer desktops over the internet incl. macOS/MLX , a one-click signed agent, and tiered cloud fallback | Balancer + agent tools Paddler, HiveCore | A balancer with self-registering pull workers | Per-tier cloud fallback and a signed desktop app built for non-technical users | The distinctive combination — federate a fleet of end-user desktops via a signed agent + tiered cloud fallback + single-tenant self-hosting, behind one OpenAI-compatible API — is what LLMrPro is for. LLMrPro/ ├── balancer/ the router: Meteor server + MongoDB + React admin UI ├── agent/ the desktop worker: Electron + React app ├── docs/ operator documentation deploy, API, agent install, troubleshooting ├── infra/ provisioning + deploy scripts Debian VM, reverse proxy, systemd ├── DESIGN.md full architecture, wire protocol, routing algorithm └── SECURITY.md security model + how to report a vulnerability LLMrPro has two installable pieces: the balancer you run one, server-side and the agent installed on each worker machine . Requirements: Node.js 22+, MongoDB 6+ a replica set is required for Meteor 3 change-stream reactivity , Meteor 3.4+, and — in production — a TLS-terminating reverse proxy nginx / Caddy / Traefik . Local / development git clone https://github.com/gysho/LLMrPro.git cd LLMrPro/balancer meteor npm install cp .env.production.example .env.production then fill in the secrets see below npm run dev inject-secrets + meteor run on :3500 Open the admin UI at http://localhost:3500/admin . Production — provision a fresh host Debian 12; installs Node, a MongoDB replica set, nginx, a systemd unit, and ufw : sudo bash infra/azure/setup-vm.sh balancer.example.com Production — deploy a release build → rsync → atomic symlink swap → restart : cd balancer ./scripts/deploy.sh https://balancer.example.com deploy@your-host curl -i https://balancer.example.com/healthz expect: 200 ok Full walkthrough: docs/operator-runbook.md /Gysho/LLMrPro/blob/main/docs/operator-runbook.md . Secrets are never committed. settings.json holds ${VAR} placeholders; real values live in .env.production and are injected into settings.runtime.json at boot by inject-secrets.sh . You provide: a platform bearer key stored only as a SHA-256 hash , an agent-token HMAC signing key, the bootstrap admin password, and one cloud API key per tier. See docs/operator-runbook.md §2 /Gysho/LLMrPro/blob/main/docs/operator-runbook.md . Each worker machine runs the agent, which connects outbound to the balancer no inbound ports and forwards jobs to a local inference engine. End users — one-line install serves the right artifact for the OS/arch and verifies its SHA-256 : curl -fsSL https://balancer.example.com/install.sh | sh Inspect it first with curl -fsSL https://balancer.example.com/install.sh.txt . On first launch the agent opens a Pairing screen — paste the balancer URL and a pairing code an admin generates these in the balancer UI , and the machine joins the pool. Per platform macOS 12+, Apple Silicon or Intel : .dmg , or the curl installer. Ships a bundled MLX inference engine; you can also point it at LM Studio / Ollama. Linux x86 64 : .AppImage primary , .deb , or .rpm . Requires a local engine LM Studio / Ollama / vLLM / llama.cpp . Manual install: sudo apt install ./