Your machines run the models. The cloud only fills the gaps.Call every model through one OpenAI format — self-hosted, and yours to own.
LLMrPro is a self-hosted LLM router. It exposes one OpenAI-compatible API and serves each request from a pool of machines you own — desktops, workstations, servers — running local inference engines. When no local worker can serve a request, it transparently falls back to the cloud provider you configured for that tier. One endpoint, one request format, your hardware first, your keys, your data.
One OpenAI-compatible API for everything. Point any OpenAI client or SDK at the balancer URL —POST /{frontier,mini,nano}/v1/chat/completions
with a standard Chat Completions body. Swap models and providers behind it without changing a line of caller code.Pools your own machines as inference workers. A lightweight desktop agent runs on each machine, connectsoutbound over a websocket (no inbound ports, no firewall changes), and pulls inference jobs. Each worker runs a full model on a local engine — LM Studio, Ollama, vLLM, llama.cpp, or the bundled MLX engine on macOS.Automatic per-tier cloud fallback. Three tiers —frontier
,mini
,nano
— each with its own cloud backend (OpenAI, Anthropic, Google, Azure OpenAI). If the local pool can't serve a tier, the request falls back to that tier's cloud provider, with configurable retry/backoff.Single-tenant and self-hosted. One balancer per organization. Your keys, your data, your infrastructure — nothing leaves your control unless a request falls back to a cloud provider you configured.Live pool management. A built-in admin UI — device pairing, agent revocation, tier config, dispatch log, and health — backed by real-time pub/sub.Security built in. Per-source rate limiting, an egress guard (SSRF protection), provider keys encrypted at rest, signed agent tokens with expiry, and a hardened install/deploy path.
OpenAI-compatible ┌───────────────────────────────┐
client / app ───────────▶ │ LLMrPro │
(Bearer key) │ balancer (router + admin) │
│ tier: frontier / mini / nano │
└───────┬────────────────┬───────┘
websocket │ dispatch │ fallback
(job pull) ▼ (local first) ▼ (per tier)
┌───────────────────────┐ ┌───────────────────┐
│ desktop agents │ │ cloud providers │
│ (your machines) │ │ OpenAI/Anthropic │
│ MLX / Ollama / vLLM │ │ / Google / Azure │
└───────────────────────┘ └───────────────────┘
- A client sends a standard OpenAI Chat Completions request to a tier endpoint with the bearer key.
- The balancer picks an eligible paired agent and dispatches the job; the agent runs it on its local engine and streams the result back.
- If no agent can serve the tier, the balancer calls that tier's configured cloud provider and
returns the result in the same OpenAI format.
X-Balancer-*
response headers report who served the request and why.
Plenty of projects touch part of this; as of 2026 no single open-source project combines all of it.
| Compared to | What they do | What LLMrPro adds |
|---|---|---|
| Distributed inference (exo, Petals) | ||
| Split one model across devices via P2P / tensor parallelism | A job-dispatch worker pool (a full model per worker) plus per-tier cloud fallback, single-tenant | |
| LLM gateways (LiteLLM, Bifrost) | ||
| Route and fall back across static, pre-configured endpoints | A live pool of your own desktops that pair and pull jobs — no static config — plus a signed end-user agent | |
| GPU-fleet managers (GPUStack, Kalavai) | ||
| Worker/coordinator scheduling for owned GPUs | Consumer desktops over the internet (incl. macOS/MLX), a one-click signed agent, and tiered cloud fallback | |
| Balancer + agent tools (Paddler, HiveCore) | ||
| A balancer with self-registering pull workers | Per-tier cloud fallback and a signed desktop app built for non-technical users |
The distinctive combination — federate a fleet of end-user desktops via a signed agent + tiered cloud fallback + single-tenant self-hosting, behind one OpenAI-compatible API — is what LLMrPro is for.
LLMrPro/
├── balancer/ # the router: Meteor server + MongoDB + React admin UI
├── agent/ # the desktop worker: Electron + React app
├── docs/ # operator documentation (deploy, API, agent install, troubleshooting)
├── infra/ # provisioning + deploy scripts (Debian VM, reverse proxy, systemd)
├── DESIGN.md # full architecture, wire protocol, routing algorithm
└── SECURITY.md # security model + how to report a vulnerability
LLMrPro has two installable pieces: the balancer (you run one, server-side) and the agent (installed on each worker machine).
Requirements: Node.js 22+, MongoDB 6+ (a replica set is required for Meteor 3 change-stream reactivity), Meteor 3.4+, and — in production — a TLS-terminating reverse proxy (nginx / Caddy / Traefik).
Local / development
git clone https://github.com/gysho/LLMrPro.git
cd LLMrPro/balancer
meteor npm install
cp .env.production.example .env.production # then fill in the secrets (see below)
npm run dev # inject-secrets + meteor run on :3500
Open the admin UI at http://localhost:3500/admin
.
Production — provision a fresh host (Debian 12; installs Node, a MongoDB replica set, nginx, a systemd unit, and ufw):
sudo bash infra/azure/setup-vm.sh balancer.example.com
Production — deploy a release (build → rsync → atomic symlink swap → restart):
cd balancer
./scripts/deploy.sh https://balancer.example.com deploy@your-host
curl -i https://balancer.example.com/healthz # expect: 200 ok
Full walkthrough: docs/operator-runbook.md.
Secrets are never committed. settings.json
holds ${VAR}
placeholders; real values live in
.env.production
and are injected into settings.runtime.json
at boot by inject-secrets.sh
. You provide: a platform bearer key (stored only as a SHA-256 hash), an agent-token HMAC signing key, the bootstrap admin password, and one cloud API key per tier. See docs/operator-runbook.md §2.
Each worker machine runs the agent, which connects outbound to the balancer (no inbound ports) and forwards jobs to a local inference engine.
End users — one-line install (serves the right artifact for the OS/arch and verifies its SHA-256):
curl -fsSL https://balancer.example.com/install.sh | sh
Inspect it first with curl -fsSL https://balancer.example.com/install.sh.txt
. On first launch the agent opens a Pairing screen — paste the balancer URL and a pairing code (an admin generates these in the balancer UI), and the machine joins the pool.
Per platform
macOS(12+, Apple Silicon or Intel):.dmg
, or the curl installer. Ships a bundled MLX inference engine; you can also point it at LM Studio / Ollama.Linux(x86_64):.AppImage
(primary),.deb
, or.rpm
. Requires a local engine (LM Studio / Ollama / vLLM / llama.cpp). Manual install:sudo apt install ./<agent>.deb
orsudo rpm -i <agent>.rpm
.- Manual downloads are listed at
https://balancer.example.com/agent/downloads
.
Build the agent from source
cd agent
npm install
npm run build:linux # .AppImage / .deb / .rpm (on a Linux host)
npm run build:mac # .dmg + bundled MLX engine (on a macOS host)
The desktop agent is distributed unsigned by design — see SECURITY.md.
curl https://balancer.example.com/frontier/v1/chat/completions \
-H "Authorization: Bearer <platform-key>" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-5",
"messages": [{ "role": "user", "content": "Hello" }]
}'
Endpoints:POST /{frontier,mini,nano}/v1/chat/completions
— the tier is the URL path.Body: standard OpenAIChat Completions. Streaming, tools, and structured outputs pass through.** Auth:**Authorization: Bearer <platform-key>
(required on every tier).- The response is OpenAI-formatted whether served by a local agent or a cloud fallback. Full contract: docs/api-contract.md.
All server config lives in balancer/settings.json
(template: balancer/settings-example.json
). Key blocks:
publicHostname
/urlSuffix
— the public origin (and reverse-proxy path prefix, if any).tiers.{frontier,mini,nano}.cloudFallback
— per-tier cloud backend:provider
(openai / anthropic / google / azure),apiSurface
,endpoint
,apiKey
,deployment
,capabilities
,parameterMap
.rateLimits
—completionsPerSecond
,pairAttemptsPerMinutePerIp
.trustProxy
— settrue
behind a trusted reverse proxy to get per-client rate limiting.egressAllowlist
— optional hostname allowlist for outbound cloud calls (SSRF hardening).cloudRetry
,healthCheck
,pairingCodeTtlMs
,dispatchTimeoutMs
,maxInflightPerAgent
.
Secrets in settings.json
are ${VAR}
placeholders resolved from the environment at boot — never commit real values.
Operator runbook— deploy, configure tiers, rotate keys, monitor, update, roll backArchitecture·full designAPI contract- Agent install: macOS·Linux Troubleshooting·FAQ
The security model, the unsigned-agent distribution tradeoff, and how to report a vulnerability are in SECURITY.md. Provider keys are encrypted at rest, agent tokens are signed with expiry, and the balancer ships with rate limiting and SSRF egress protection.
LLMrPro is released under the MIT License. © 2026 Gysho.