Show HN: A minimalist proxy for your local LLM cluster (~1100 lines) A developer released smol-llm-proxy, a minimalist API proxy for self-hosted llama.cpp setups that routes across multiple llama-server instances with per-user API keys and token usage tracking, in approximately 1100 lines of code using about 53 MB of RAM. The proxy includes in-memory rate limiting with async SQLite flush and is designed for users running multiple llama-server instances on different models or GPUs who need to share them with a small group while tracking usage. A small API proxy for self-hosted llama.cpp setups. Routes across multiple llama-server instances, per-user API keys, token usage tracking. In-memory rate limiting RPM/TPM with async SQLite flush. ~0.13 ms is proxy logic per-request , the 2–4 ms end-to-end figure includes the FastAPI/uvicorn/httpx baseline and asyncio event-loop contention at high concurrency. <=1100 code lines, ~53 MB RAM. Built for the case where you run multiple llama-server instances different models, different GPUs and want to share them across users with token tracking. smol-llm-proxy is built for one specific case: multiple llama-server instances, multiple users, per-user token accounting . — much broader scope: 100+ cloud providers, virtual keys, budgets, admin UI, fallbacks. Requires Postgres + Redis for full features. Use it if you need a production gateway across cloud LLMs. LiteLLM https://github.com/BerriAI/litellm — solves a different problem: hot-swapping models on one llama.cpp instance. No users, no accounting. Use it if you run many models on one machine and want them loaded on demand. llama-swap https://github.com/mostlygeek/llama-swap — built into llama-server itself. Same scope as llama-swap, no auth layer. llama.cpp router mode https://github.com/ggerganov/llama.cpp If you self-host several llama-server instances on one or more machines and want to share them with a small group while tracking usage, smol-llm-proxy is the smallest thing that does that. Otherwise, one of the above is probably a better fit. users ──HTTP── proxy :port ──HTTP── llama-server 1 :port │ llama-server 2 :port │ llama-server N :port │ ├── in-memory cache keys, routes — TTL 30s ├── validate API key + resolve routing SQLite on first call, then cache ├── rate limiter: read-only DB check + in-memory reservation + reconcile on response ├── async batch flush to SQLite 1s interval ├── forward request via connection-pooled httpx client ├── async usage logger batch queue, 50 items / 1s timeout └── retention cleanup 90-day purge, daily TLS termination is handled externally — Cloudflare, Caddy, nginx, or any reverse proxy in front of the proxy. - Per-user API keys create / delete / toggle active Rate limiting — per-key RPM/TPM, sliding 60-second window, 429 + Retry-After on exceed Multi-server routing by model name with in-memory cache TTL 30s Model aliases alias - model-name.gguf Token usage logging — prompt/completion tokens, timings, 90-day retention with daily purge Streaming and non-streaming proxy support for chat, completions, and embeddings Connection-pooled httpx client keepalive connections to backends SQLite backend zero external DB dependencies Clone the repo, then: cp .env.example .env set ADMIN KEY cp config.example.yaml config.yaml fill in your servers docker compose up -d --build The proxy listens on 0.0.0.0:8000 by default. docker build -t smol-llm-proxy . docker run -p 8000:8000 \ -e ADMIN KEY=secret \ -v db-data:/app/data \ -v $ pwd /config.yaml:/app/config.yaml:ro \ smol-llm-proxy pip install smol-llm-proxy Example configs ship with the package. Copy them: python python -c "import smol llm proxy, shutil, os; d=os.path.dirname smol llm proxy. file ; shutil.copy2 f'{d}/config.example.yaml','config.yaml' ; shutil.copy2 f'{d}/.env.example','.env' " Edit .env and set ADMIN KEY, then uncomment and fill in config.yaml with your servers ADMIN KEY=secret smol-llm-proxy or: ADMIN KEY=secret python -m smol llm proxy Or download from GitHub: curl -sO https://raw.githubusercontent.com/robolamp/smol-llm-proxy/main/.env.example && cp .env.example .env curl -sO https://raw.githubusercontent.com/robolamp/smol-llm-proxy/main/config.example.yaml && cp config.example.yaml config.yaml git clone https://github.com/robolamp/smol-llm-proxy.git cd smol-llm-proxy pip install -e . Copy example configs and run: cp .env.example .env set ADMIN KEY cp config.example.yaml config.yaml fill in your servers ADMIN KEY=secret python -m smol llm proxy - Create a user key: curl -X POST http://localhost:8000/admin/keys \ -H "Authorization: Bearer $ADMIN KEY" \ -d '{"name": "my-user"}' The response contains a JSON object with a key field — that's the user's Bearer token. Save it; you'll need it for proxy requests. - Send a chat completion with the user key: curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-