# Free LLM balancer combines multiple local inference machines with cloud fallback

> Source: <https://github.com/Gysho/LLMrPro>
> Published: 2026-07-20 21:24:57+00:00

Your machines run the models. The cloud only fills the gaps.Call every model through one OpenAI format — self-hosted, and yours to own.

LLMrPro is a self-hosted LLM router. It exposes **one OpenAI-compatible API** and serves each
request from a pool of machines **you** own — desktops, workstations, servers — running local
inference engines. When no local worker can serve a request, it transparently falls back to the
cloud provider you configured for that tier. One endpoint, one request format, your hardware
first, your keys, your data.

**One OpenAI-compatible API for everything.** Point any OpenAI client or SDK at the balancer URL —`POST /{frontier,mini,nano}/v1/chat/completions`

with a standard Chat Completions body. Swap models and providers behind it without changing a line of caller code.**Pools your own machines as inference workers.** A lightweight desktop agent runs on each machine, connects**outbound** over a websocket (no inbound ports, no firewall changes), and pulls inference jobs. Each worker runs a full model on a local engine — LM Studio, Ollama, vLLM, llama.cpp, or the bundled MLX engine on macOS.**Automatic per-tier cloud fallback.** Three tiers —`frontier`

,`mini`

,`nano`

— each with its own cloud backend (OpenAI, Anthropic, Google, Azure OpenAI). If the local pool can't serve a tier, the request falls back to that tier's cloud provider, with configurable retry/backoff.**Single-tenant and self-hosted.** One balancer per organization. Your keys, your data, your infrastructure — nothing leaves your control unless a request falls back to a cloud provider you configured.**Live pool management.** A built-in admin UI — device pairing, agent revocation, tier config, dispatch log, and health — backed by real-time pub/sub.**Security built in.** Per-source rate limiting, an egress guard (SSRF protection), provider keys encrypted at rest, signed agent tokens with expiry, and a hardened install/deploy path.

```
   OpenAI-compatible          ┌───────────────────────────────┐
   client / app  ───────────▶ │            LLMrPro             │
   (Bearer key)               │   balancer  (router + admin)   │
                              │  tier: frontier / mini / nano  │
                              └───────┬────────────────┬───────┘
                        websocket     │ dispatch        │ fallback
                        (job pull)    ▼ (local first)   ▼ (per tier)
                       ┌───────────────────────┐   ┌───────────────────┐
                       │  desktop agents        │   │  cloud providers  │
                       │  (your machines)       │   │  OpenAI/Anthropic │
                       │  MLX / Ollama / vLLM   │   │  / Google / Azure │
                       └───────────────────────┘   └───────────────────┘
```

- A client sends a standard OpenAI Chat Completions request to a tier endpoint with the bearer key.
- The balancer picks an eligible paired agent and dispatches the job; the agent runs it on its local engine and streams the result back.
- If no agent can serve the tier, the balancer calls that tier's configured cloud provider and
returns the result in the same OpenAI format.
`X-Balancer-*`

response headers report who served the request and why.

Plenty of projects touch part of this; as of 2026 no single open-source project combines all of it.

| Compared to | What they do | What LLMrPro adds |
|---|---|---|
Distributed inference (exo, Petals) |
Split one model across devices via P2P / tensor parallelism | A job-dispatch worker pool (a full model per worker) plus per-tier cloud fallback, single-tenant |
LLM gateways (LiteLLM, Bifrost) |
Route and fall back across static, pre-configured endpoints | A live pool of your own desktops that pair and pull jobs — no static config — plus a signed end-user agent |
GPU-fleet managers (GPUStack, Kalavai) |
Worker/coordinator scheduling for owned GPUs | Consumer desktops over the internet (incl. macOS/MLX), a one-click signed agent, and tiered cloud fallback |
Balancer + agent tools (Paddler, HiveCore) |
A balancer with self-registering pull workers | Per-tier cloud fallback and a signed desktop app built for non-technical users |

The distinctive combination — **federate a fleet of end-user desktops via a signed agent +
tiered cloud fallback + single-tenant self-hosting, behind one OpenAI-compatible API** — is what
LLMrPro is for.

```
LLMrPro/
├── balancer/     # the router: Meteor server + MongoDB + React admin UI
├── agent/        # the desktop worker: Electron + React app
├── docs/         # operator documentation (deploy, API, agent install, troubleshooting)
├── infra/        # provisioning + deploy scripts (Debian VM, reverse proxy, systemd)
├── DESIGN.md     # full architecture, wire protocol, routing algorithm
└── SECURITY.md   # security model + how to report a vulnerability
```

LLMrPro has two installable pieces: the **balancer** (you run one, server-side) and the
**agent** (installed on each worker machine).

**Requirements:** Node.js 22+, MongoDB 6+ (a replica set is required for Meteor 3 change-stream
reactivity), Meteor 3.4+, and — in production — a TLS-terminating reverse proxy
(nginx / Caddy / Traefik).

**Local / development**

```
git clone https://github.com/gysho/LLMrPro.git
cd LLMrPro/balancer
meteor npm install
cp .env.production.example .env.production   # then fill in the secrets (see below)
npm run dev                                  # inject-secrets + meteor run on :3500
```

Open the admin UI at `http://localhost:3500/admin`

.

**Production — provision a fresh host** (Debian 12; installs Node, a MongoDB replica set, nginx,
a systemd unit, and ufw):

```
sudo bash infra/azure/setup-vm.sh balancer.example.com
```

**Production — deploy a release** (build → rsync → atomic symlink swap → restart):

```
cd balancer
./scripts/deploy.sh https://balancer.example.com deploy@your-host
curl -i https://balancer.example.com/healthz   # expect: 200 ok
```

Full walkthrough: [docs/operator-runbook.md](/Gysho/LLMrPro/blob/main/docs/operator-runbook.md).

**Secrets** are never committed. `settings.json`

holds `${VAR}`

placeholders; real values live in
`.env.production`

and are injected into `settings.runtime.json`

at boot by `inject-secrets.sh`

.
You provide: a platform bearer key (stored only as a SHA-256 hash), an agent-token HMAC signing
key, the bootstrap admin password, and one cloud API key per tier. See
[docs/operator-runbook.md §2](/Gysho/LLMrPro/blob/main/docs/operator-runbook.md).

Each worker machine runs the agent, which connects outbound to the balancer (no inbound ports) and forwards jobs to a local inference engine.

**End users — one-line install** (serves the right artifact for the OS/arch and verifies its
SHA-256):

```
curl -fsSL https://balancer.example.com/install.sh | sh
```

Inspect it first with `curl -fsSL https://balancer.example.com/install.sh.txt`

. On first launch
the agent opens a **Pairing** screen — paste the balancer URL and a pairing code (an admin
generates these in the balancer UI), and the machine joins the pool.

**Per platform**

**macOS**(12+, Apple Silicon or Intel):`.dmg`

, or the curl installer. Ships a bundled MLX inference engine; you can also point it at LM Studio / Ollama.**Linux**(x86_64):`.AppImage`

(primary),`.deb`

, or`.rpm`

. Requires a local engine (LM Studio / Ollama / vLLM / llama.cpp). Manual install:`sudo apt install ./<agent>.deb`

or`sudo rpm -i <agent>.rpm`

.- Manual downloads are listed at
`https://balancer.example.com/agent/downloads`

.

**Build the agent from source**

```
cd agent
npm install
npm run build:linux      # .AppImage / .deb / .rpm   (on a Linux host)
npm run build:mac        # .dmg + bundled MLX engine (on a macOS host)
```

The desktop agent is distributed **unsigned** by design — see [SECURITY.md](/Gysho/LLMrPro/blob/main/SECURITY.md).

```
curl https://balancer.example.com/frontier/v1/chat/completions \
  -H "Authorization: Bearer <platform-key>" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-5",
        "messages": [{ "role": "user", "content": "Hello" }]
      }'
```

**Endpoints:**`POST /{frontier,mini,nano}/v1/chat/completions`

— the tier is the URL path.**Body:** standard OpenAI**Chat Completions**. Streaming, tools, and structured outputs pass through.** Auth:**`Authorization: Bearer <platform-key>`

(required on every tier).- The response is OpenAI-formatted whether served by a local agent or a cloud fallback. Full
contract:
[docs/api-contract.md](/Gysho/LLMrPro/blob/main/docs/api-contract.md).

All server config lives in `balancer/settings.json`

(template: `balancer/settings-example.json`

).
Key blocks:

`publicHostname`

/`urlSuffix`

— the public origin (and reverse-proxy path prefix, if any).`tiers.{frontier,mini,nano}.cloudFallback`

— per-tier cloud backend:`provider`

(openai / anthropic / google / azure),`apiSurface`

,`endpoint`

,`apiKey`

,`deployment`

,`capabilities`

,`parameterMap`

.`rateLimits`

—`completionsPerSecond`

,`pairAttemptsPerMinutePerIp`

.`trustProxy`

— set`true`

behind a trusted reverse proxy to get per-client rate limiting.`egressAllowlist`

— optional hostname allowlist for outbound cloud calls (SSRF hardening).`cloudRetry`

,`healthCheck`

,`pairingCodeTtlMs`

,`dispatchTimeoutMs`

,`maxInflightPerAgent`

.

Secrets in `settings.json`

are `${VAR}`

placeholders resolved from the environment at boot —
never commit real values.

[Operator runbook](/Gysho/LLMrPro/blob/main/docs/operator-runbook.md)— deploy, configure tiers, rotate keys, monitor, update, roll back[Architecture](/Gysho/LLMrPro/blob/main/docs/architecture.md)·[full design](/Gysho/LLMrPro/blob/main/DESIGN.md)[API contract](/Gysho/LLMrPro/blob/main/docs/api-contract.md)- Agent install:
[macOS](/Gysho/LLMrPro/blob/main/docs/agent-install-guide-mac.md)·[Linux](/Gysho/LLMrPro/blob/main/docs/agent-install-guide-linux.md) [Troubleshooting](/Gysho/LLMrPro/blob/main/docs/troubleshooting.md)·[FAQ](/Gysho/LLMrPro/blob/main/docs/faq.md)

The security model, the unsigned-agent distribution tradeoff, and how to report a vulnerability
are in [SECURITY.md](/Gysho/LLMrPro/blob/main/SECURITY.md). Provider keys are encrypted at rest, agent tokens are signed
with expiry, and the balancer ships with rate limiting and SSRF egress protection.

LLMrPro is released under the [MIT License](/Gysho/LLMrPro/blob/main/LICENSE). © 2026 Gysho.