{"slug": "show-hn-a-minimalist-proxy-for-your-local-llm-cluster-1100-lines", "title": "Show HN: A minimalist proxy for your local LLM cluster (~1100 lines)", "summary": "A developer released smol-llm-proxy, a minimalist API proxy for self-hosted llama.cpp setups that routes across multiple llama-server instances with per-user API keys and token usage tracking, in approximately 1100 lines of code using about 53 MB of RAM. The proxy includes in-memory rate limiting with async SQLite flush and is designed for users running multiple llama-server instances on different models or GPUs who need to share them with a small group while tracking usage.", "body_md": "A small API proxy for self-hosted llama.cpp setups. Routes across multiple llama-server instances, per-user API keys, token usage tracking. In-memory rate limiting (RPM/TPM) with async SQLite flush. ~0.13 ms is proxy logic (per-request), the 2–4 ms end-to-end figure includes the FastAPI/uvicorn/httpx baseline and asyncio event-loop contention at high concurrency. <=1100 code lines, ~53 MB RAM.\n\nBuilt for the case where you run multiple llama-server instances (different models, different GPUs) and want to share them across users with token tracking.\n\nsmol-llm-proxy is built for one specific case: **multiple llama-server instances, multiple users, per-user token accounting**.\n\n— much broader scope: 100+ cloud providers, virtual keys, budgets, admin UI, fallbacks. Requires Postgres + Redis for full features. Use it if you need a production gateway across cloud LLMs.[LiteLLM](https://github.com/BerriAI/litellm)— solves a different problem: hot-swapping models on one llama.cpp instance. No users, no accounting. Use it if you run many models on one machine and want them loaded on demand.[llama-swap](https://github.com/mostlygeek/llama-swap)— built into llama-server itself. Same scope as llama-swap, no auth layer.[llama.cpp router mode](https://github.com/ggerganov/llama.cpp)\n\nIf you self-host several llama-server instances on one or more machines and want to share them with a small group while tracking usage, smol-llm-proxy is the smallest thing that does that. Otherwise, one of the above is probably a better fit.\n\n```\n[users] ──HTTP──> [proxy :port] ──HTTP──> [llama-server 1 :port]\n                       │                  [llama-server 2 :port]\n                       │                  [llama-server N :port]\n                       │\n                       ├── in-memory cache (keys, routes) — TTL 30s\n                       ├── validate API key + resolve routing (SQLite on first call, then cache)\n                       ├── rate limiter: read-only DB check + in-memory reservation + reconcile on response\n                       ├── async batch flush to SQLite (1s interval)\n                       ├── forward request via connection-pooled httpx client\n                       ├── async usage logger (batch queue, 50 items / 1s timeout)\n                       └── retention cleanup (90-day purge, daily)\n```\n\nTLS termination is handled externally — Cloudflare, Caddy, nginx, or any reverse proxy in front of the proxy.\n\n- Per-user API keys (create / delete / toggle active)\n**Rate limiting**— per-key RPM/TPM, sliding 60-second window, 429 +`Retry-After`\n\non exceed**Multi-server routing** by model name with in-memory cache (TTL 30s)**Model aliases**(`alias`\n\n->`model-name.gguf`\n\n)**Token usage logging**— prompt/completion tokens, timings, 90-day retention with daily purge** Streaming and non-streaming**proxy support for chat, completions, and embeddings** Connection-pooled httpx client**(keepalive connections to backends)** SQLite backend**(zero external DB dependencies)\n\nClone the repo, then:\n\n```\ncp .env.example .env                # set ADMIN_KEY\ncp config.example.yaml config.yaml  # fill in your servers\ndocker compose up -d --build\n```\n\nThe proxy listens on `0.0.0.0:8000`\n\nby default.\n\n```\ndocker build -t smol-llm-proxy .\ndocker run -p 8000:8000 \\\n  -e ADMIN_KEY=secret \\\n  -v db-data:/app/data \\\n  -v $(pwd)/config.yaml:/app/config.yaml:ro \\\n  smol-llm-proxy\npip install smol-llm-proxy\n```\n\nExample configs ship with the package. Copy them:\n\n``` python\npython -c \"import smol_llm_proxy, shutil, os; d=os.path.dirname(smol_llm_proxy.__file__); shutil.copy2(f'{d}/config.example.yaml','config.yaml'); shutil.copy2(f'{d}/.env.example','.env')\"\n# Edit .env and set ADMIN_KEY, then uncomment and fill in config.yaml with your servers\nADMIN_KEY=secret smol-llm-proxy\n# or: ADMIN_KEY=secret python -m smol_llm_proxy\n```\n\nOr download from GitHub:\n\n```\ncurl -sO https://raw.githubusercontent.com/robolamp/smol-llm-proxy/main/.env.example && cp .env.example .env\ncurl -sO https://raw.githubusercontent.com/robolamp/smol-llm-proxy/main/config.example.yaml && cp config.example.yaml config.yaml\ngit clone https://github.com/robolamp/smol-llm-proxy.git\ncd smol-llm-proxy\npip install -e .\n```\n\nCopy example configs and run:\n\n```\ncp .env.example .env                # set ADMIN_KEY\ncp config.example.yaml config.yaml  # fill in your servers\nADMIN_KEY=secret python -m smol_llm_proxy\n```\n\n- Create a user key:\n\n```\ncurl -X POST http://localhost:8000/admin/keys \\\n  -H \"Authorization: Bearer $ADMIN_KEY\" \\\n  -d '{\"name\": \"my-user\"}'\n```\n\nThe response contains a JSON object with a `key`\n\nfield — that's the user's Bearer token. Save it; you'll need it for proxy requests.\n\n- Send a chat completion with the user key:\n\n```\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer sk-<full-key-from-step-1>\" \\\n  -d '{\n    \"model\": \"Qwen3.5-2B\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"stream\": true\n  }'\n```\n\nThe proxy reads two files: `config.yaml`\n\nfor routing and `.env`\n\nfor runtime settings.\n\nLoaded into SQLite at startup, persisted across restarts:\n\n```\nservers:\n  - name: my-server\n    url: http://host:port\n    api_key: \"\"              # optional, if llama-server requires auth\n    models:\n      - model-name.gguf\n\naliases:\n  alias: model-name.gguf     # short name -> real model name\n```\n\nOn startup, `config.yaml`\n\nis merged into the database:\n\n- Servers/aliases\n**present** in`config.yaml`\n\nare**upserted** into the database (new ones created, existing ones updated). - Servers/aliases present in the database but\n**not** in`config.yaml`\n\nare**kept**— YAML is additive, not destructive. - Models listed under a server in YAML are synced; extra models on that server are removed.\n- API keys, usage logs, and rate limits are\n**not** affected by config sync. - The in-memory route cache is cleared after sync.\n\nServers and aliases managed via the Admin API persist across restarts. To remove them, use the Admin API (`DELETE /admin/servers/{id}`\n\n, `DELETE /admin/aliases/{name}`\n\n).\n\nThe Compose setup mounts two volumes:\n\n`${DATA_DIR:-./data}:/app/data`\n\n— SQLite database (`DB_PATH=/app/data/proxy.db`\n\n), persists across container restarts. Override with`DATA_DIR`\n\nin`.env`\n\n.`./config.yaml:/config/config.yaml:ro`\n\n— config file, read-only. Set`CONFIG_PATH=/config/config.yaml`\n\nin Compose (hardcoded, not from`.env`\n\n).\n\n## Reference: Admin API\n\nAll admin endpoints require `Authorization: Bearer <ADMIN_KEY>`\n\nheader.\n\n| Endpoint | Method | Description |\n|---|---|---|\n`/admin/keys` |\n`GET` / `POST` |\nList API keys / Create user key |\n`/admin/keys/{key_id}` |\n`DELETE` |\nRevoke key (integer id, not key string) |\n`/admin/keys/{key_id}/toggle` |\n`PATCH` |\nActivate/deactivate key |\n`/admin/keys/{key_id}/limits` |\n`PUT` |\nSet per-key RPM/TPM rate limits |\n`/admin/servers` |\n`GET` / `POST` |\nList servers / Register a llama-server |\n`/admin/servers/{server_id}` |\n`DELETE` / `PATCH` |\nRemove or update server |\n`/admin/servers/{server_id}/models` |\n`POST` |\nAssign model to server (use `?reassign=true` to move from another server) |\n`/admin/servers/{server_id}/models/{model_name}` |\n`DELETE` |\nUnassign model from server |\n`/admin/aliases` |\n`GET` / `POST` |\nList aliases / Create alias |\n`/admin/aliases/{alias_name}` |\n`PATCH` / `DELETE` |\nUpdate alias target / Delete alias |\n`/admin/usage` |\n`GET` |\nToken usage logs + summary (accepts `key_id` , `server_id` , `start_date` , `end_date` , `limit` , `offset` ) |\n`/admin/usage/summary/real` |\n`GET` |\nUsage summary grouped by real model name + server |\n\n**Note:** Key operations (`DELETE`\n\n, `PATCH /toggle`\n\n, `PUT /limits`\n\n) use integer `key_id`\n\nfrom the database, not the API key string itself.\n\nDefault limits: **100 RPM**, **50,000 TPM**. Set custom limits per key:\n\n```\ncurl -X PUT http://localhost:8000/admin/keys/1/limits \\\n  -H \"Authorization: Bearer $ADMIN_KEY\" \\\n  -d '{\"rpm_limit\": 200, \"tpm_limit\": 100000}'\n```\n\nExceeding limits returns `429 Too Many Requests`\n\nwith a `Retry-After`\n\nheader. The rate limiter reads from SQLite + in-memory counters on every check and commits in-memory with async batch flush — one read-only SELECT per request, no commit overhead. **Note:** rate counters and caches are process-local; under `--workers N`\n\neach worker maintains independent counters (limits multiply by N).\n\nEach model can be assigned to **exactly one server** at a time. Assigning a model that is already on another server returns `409 Conflict`\n\n:\n\n```\ncurl -X POST http://localhost:8000/admin/servers/2/models \\\n  -H \"Authorization: Bearer $ADMIN_KEY\" \\\n  -d '{\"model_name\": \"qwen3-30b-a3b-instruct.gguf\"}'\n# 409: Model 'qwen3-30b-a3b-instruct.gguf' already assigned to server 'server-a' (id=1). Use ?reassign=true to move.\n```\n\nMove a model to a different server with `?reassign=true`\n\n:\n\n```\ncurl -X POST \"http://localhost:8000/admin/servers/2/models?reassign=true\" \\\n  -H \"Authorization: Bearer $ADMIN_KEY\" \\\n  -d '{\"model_name\": \"qwen3-30b-a3b-instruct.gguf\"}'\n```\n\nMultiple aliases can point to the same model. Aliases can be switched to a different model via `PATCH`\n\n:\n\n```\ncurl -X PATCH http://localhost:8000/admin/aliases/fast \\\n  -H \"Authorization: Bearer $ADMIN_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"real_model_name\": \"new-model.gguf\"}'\n```\n\nView raw usage logs:\n\n```\ncurl \"http://localhost:8000/admin/usage?key_id=1&start_date=2024-01-01T00:00:00&end_date=2024-01-31T23:59:59\" \\\n  -H \"Authorization: Bearer $ADMIN_KEY\"\n```\n\nUsage summary grouped by real model name + server (aggregates across all aliases):\n\n```\ncurl \"http://localhost:8000/admin/usage/summary/real?key_id=1&start_date=2024-01-01T00:00:00&end_date=2024-01-31T23:59:59\" \\\n  -H \"Authorization: Bearer $ADMIN_KEY\"\n```\n\nAll endpoints accept: `key_id`\n\n, `server_id`\n\n, `start_date`\n\n(ISO 8601), `end_date`\n\n(ISO 8601), `limit`\n\n, `offset`\n\n. Usage logs are preserved even after a server or key is deleted.\n\n## Reference: Proxy Endpoints\n\nThese forward to llama-server backends based on model name routing.\n\n| Endpoint | Method | Description |\n|---|---|---|\n`/v1/chat/completions` |\n`POST` |\nChat completions (streaming + non-streaming) |\n`/v1/completions` |\n`POST` |\nLegacy completions |\n`/v1/embeddings` |\n`POST` |\nEmbeddings |\n`/v1/models` |\n`GET` |\nList available models (requires user API key — returns 401 without Authorization, 403 for invalid/inactive key, fans out to all backends) |\n`/health` |\n`GET` |\nHealth check (no auth required) |\n\nEach request logs: user, server, model name, prompt/completion tokens, timings (ms), and total tokens. No conversation content is stored.\n\n## Reference: Environment Variables\n\n| Variable | Default | Description |\n|---|---|---|\n`ADMIN_KEY` |\nrequired |\nBearer token for `/admin/*` endpoints. Startup fails without it. |\n`PROXY_HOST` |\n`0.0.0.0` |\nListen address |\n`PROXY_PORT` |\n`8000` |\nListen port |\n`DB_PATH` |\n`data/proxy.db` |\nSQLite database location |\n`CONFIG_PATH` |\n`config.yaml` |\nPath to YAML config file |\n`HTTPX_TIMEOUT` |\n`120` |\nUpstream HTTP client timeout (seconds) |\n`PROXY_MAX_CONNECTIONS` |\n`50` |\nhttpx connection pool max size |\n`PROXY_MAX_KEEPALIVE` |\n`20` |\nhttpx keepalive pool max size |\n`PROXY_KEEPALIVE_EXPIRY` |\n`30.0` |\nhttpx keepalive expiry (seconds) |\n`BENCH_COLD_CACHE` |\n(unset) |\nSet to `\"1\"` to disable in-memory caches (for benchmarking) |\n\nFor `pip install`\n\n, set them in shell or via a `.env`\n\n-like loader (the proxy reads `.env`\n\nfrom the current working directory at startup). For Docker Compose, put them in `.env`\n\n:\n\n```\nADMIN_KEY=secret\nPROXY_PORT=8000\n```\n\n## Reference: Dependencies\n\n| Package | Version | Notes |\n|---|---|---|\n`fastapi` |\n>= 0.115.0 | Web framework |\n`uvicorn[standard]` |\n>= 0.30.0 | ASGI server |\n`httpx` |\n>= 0.27.0 | Async HTTP client |\n`pydantic` |\n>= 2.0.0 | Data validation |\n`pyyaml` |\n>= 6.0 | Config parsing |\n`orjson` |\n>= 3.9.0 | Fast JSON serialization |\n`uvloop` |\n>= 0.19.0 | Linux/macOS only (`sys_platform != 'win32'` ) |\n\nOptional dev dependencies: `pytest`\n\n, `pytest-rerunfailures`\n\n, `ruff`\n\n(see `pyproject.toml`\n\n).\n\n## Benchmarking\n\nProxy overhead measured with Locust using **parallel concurrent execution** — both benchmarks (direct and proxy) hit the same backend simultaneously for a fair comparison.\n\nAll requests authenticated, routed, and logged via SQLite on every call (cold cache mode). In-memory cache disabled to measure worst-case per-request overhead.\n\n**Hardware:** i9-14900K, RTX 4090 | **Model:** Qwen3.5-2B-GGUF | **Backend:** llama.cpp server\n\n| Benchmark | Mean Overhead | P99 Overhead | RPS Delta |\n|---|---|---|---|\n| Mock (low load) | +2 ms | +10 ms | -1.0 |\n| Mock (medium load) | +2 ms | +10 ms | -3.8 |\n| Mock (high load) | +4 ms | +10 ms | -29.0 |\n| Real (low load) | +1 ms | +20 ms | 0.0 |\n\n## Mock backend (fixed 100ms delay, clean conditions)\n\n| Mode | Users | Direct P50 | Proxy P50 | Overhead P50 | Direct P95 | Proxy P95 | Overhead P95 | Direct P99 | Proxy P99 | Overhead P99 | Direct Mean | Through proxy | Overhead Mean | Direct RPS | Through proxy | RPS overhead |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| Low | 5+5 | 100ms | 100ms | +0ms | 100ms | 100ms | +0ms | 100ms | 110ms | +10ms | 101ms | 104ms | +2ms | 49.1 | 48.1 | -1.0 |\n| Medium | 20+20 | 100ms | 100ms | +0ms | 100ms | 110ms | +10ms | 100ms | 110ms | +10ms | 102ms | 104ms | +2ms | 194.5 | 190.7 | -3.8 |\n| High | 100+100 | 100ms | 110ms | +10ms | 110ms | 110ms | +0ms | 110ms | 120ms | +10ms | 103ms | 106ms | +4ms | 893.7 | 864.7 | -29.0 |\n\n## Real llama-server backend\n\n| Mode | Users | Direct P50 | Proxy P50 | Overhead P50 | Direct P95 | Proxy P95 | Overhead P95 | Direct P99 | Proxy P99 | Overhead P99 | Direct Mean | Through proxy | Overhead Mean | Direct RPS | Through proxy | RPS overhead |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| Low | 5+5 | 430ms | 430ms | +0ms | 760ms | 750ms | -10ms | 850ms | 870ms | +20ms | 460ms | 461ms | +1ms | 10.8 | 10.8 | 0.0 |\n\nProxy overhead on clean conditions (mock): **~2–4 ms** mean, **+0–10 ms** P99 across all load levels with DB-backed rate limiter. Against real backend: **negligible at low load** (+1 ms mean), higher load limited by SQLite I/O under concurrent bench runs.\n\nRun your own benchmarks: `python tests/benchmark/run.py [low|medium|high]`\n\n(add `--mock`\n\nfor fixed-delay backend)\n\n| Workers | Idle | Under load | Growth |\n|---|---|---|---|\n| 1 | 53 MB | 62 MB | +9 MB |\n| 4 | 252 MB | 273 MB | +21 MB |\n\nPer-worker baseline: **~53 MB**, load growth: **+4–6 MB** per worker.\n\nIdentical footprint against mock and real backends — the proxy forwards without buffering responses. No memory growth observed across extended runs.\n\nThe aggregate overhead numbers (+2–4 ms mean, +0–10 ms P99 on mock) include asyncio event loop contention at high concurrency (100+ concurrent users per worker). Per-request proxy logic itself is **~0.13 ms** with uvloop — the difference comes from how asyncio handles many simultaneous awaits on a single thread. With 4 uvicorn workers, each worker handles ~25 requests, keeping contention minimal.\n\nReal backend P99 spikes (Medium mode, ~14 s) are caused by llama.cpp single-thread inference bottleneck under 20 concurrent users — not proxy overhead. Proxy adds negligible latency regardless of backend saturation.", "url": "https://wpnews.pro/news/show-hn-a-minimalist-proxy-for-your-local-llm-cluster-1100-lines", "canonical_source": "https://github.com/robolamp/smol-llm-proxy", "published_at": "2026-07-30 10:56:54+00:00", "updated_at": "2026-07-30 11:22:48.450911+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["smol-llm-proxy", "llama.cpp", "LiteLLM", "llama-swap", "FastAPI", "uvicorn", "httpx", "SQLite"], "alternates": {"html": "https://wpnews.pro/news/show-hn-a-minimalist-proxy-for-your-local-llm-cluster-1100-lines", "markdown": "https://wpnews.pro/news/show-hn-a-minimalist-proxy-for-your-local-llm-cluster-1100-lines.md", "text": "https://wpnews.pro/news/show-hn-a-minimalist-proxy-for-your-local-llm-cluster-1100-lines.txt", "jsonld": "https://wpnews.pro/news/show-hn-a-minimalist-proxy-for-your-local-llm-cluster-1100-lines.jsonld"}}