One port, many models: A tiny, stateless HTTP router for OpenAI-compatible LLM inference backends.
LLMhop peeks at the model
field of an incoming OpenAI-compatible request and reverse-proxies it to the matching backend. It is primarily designed for single-model inference servers like vLLM and sglang that serve one model per process and need a thin model-aware gateway in front of them, but it works with any OpenAI-compatible backend (including multi-model servers and hosted providers) whenever you want to consolidate several upstreams behind a single endpoint.
-
OpenAI-compatible reverse proxy, model router and request dispatcher for self-hosted LLM inference.
-
Stateless single-binary HTTP service: no database, no cache, no background workers, safe behind any load balancer.
-
Zero external dependencies: pure Go, no third-party packages, no CGO.
-
Works with any OpenAI API-compatible backend, self-hosted or remote: vLLM, sglang, TabbyAPI, Aphrodite, Ollama, LocalAI, OpenRouter, together.ai, DeepInfra, etc.
-
Ships as a static binary, a minimal Docker image and a hardened NixOS module that can optionally spin up llama.cpp, sglang or vLLM workers alongside the router.
-
Client sends a request with a JSON body containing
{"model": "..."}
. - LLMhop reads the
model
field and looks it up in its config. - The request is forwarded verbatim to the configured backend URL.
- Unknown models return
404
.
LLMhop can optionally gate incoming requests with a list of bearer tokens and inject per-model Authorization
(or any other) headers when forwarding to the backend.
Both sides are opt-in: leave authTokens
and models.*.headers
unset and headers are forwarded verbatim.
When authTokens
is set, the router validates the incoming Authorization: Bearer <token>
header (constant-time compare) and then strips it before forwarding, so the client-facing token never leaks upstream.
Per-model headers are applied last, so a configured Authorization
always wins over whatever the client sent.
Create a config.json
:
{
"listen": ":8080",
"authTokens": ["${file:client_token}"],
"models": {
"llama-3-8b": {
"url": "http://localhost:30000"
},
"openai-gpt-4o": {
"url": "https://api.openai.com",
"headers": {
"Authorization": "Bearer ${env:OPENAI_KEY}"
}
}
}
}
String values inside authTokens
and models.*.headers
are expanded at startup, so no plaintext secret ever has to live in the config file:
${env:NAME}
: read from theNAME
environment variable.${file:path}
: read from a file. Relative paths are resolved against$CREDENTIALS_DIRECTORY
when set (e.g. when launched by systemd withLoadCredential=
), otherwise against the current working directory. A single trailing newline is trimmed.$NAME
: shorthand for${env:NAME}
.
Unresolved references are a hard startup error.
LLMhop buffers each request body in memory so it can peek at the model
field before forwarding.
To keep a single request from exhausting memory, the body is capped at 100 MiB by default; bodies beyond the cap are rejected with 413 Request Entity Too Large
. Override it when vision or other multimodal payloads need more:
{ "maxBodyBytes": 524288000 }
llmhop --config config.json
nix run github:mirkolenz/llmhop -- --config config.json
docker run --rm -p 8080:8080 -v ./config.json:/config.json ghcr.io/mirkolenz/llmhop --config /config.json
A hardened systemd service is provided out of the box. Add LLMhop to your flake inputs and import the module into your system configuration:
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
llmhop = {
url = "github:mirkolenz/llmhop";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs =
{ nixpkgs, llmhop, ... }:
{
nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
llmhop.nixosModules.default
{
services.llmhop = {
enable = true;
settings = {
listen = ":8080";
models = {
"llama-3-8b".url = "http://localhost:30000";
"qwen-2.5-7b".url = "http://localhost:30001";
};
};
};
}
];
};
};
}
The unit runs under DynamicUser
with aggressive sandboxing (ProtectSystem
, PrivateTmp
, restricted syscalls and address families, no new privileges, ...) and restarts on failure.
The module can also run the inference servers themselves, so you don't have to wire up llama.cpp, sglang or vLLM by hand.
Each backend exposes a models
attrset under services.llmhop.<backend>
and every entry becomes one isolated worker bound to a loopback port, with the matching route registered automatically with llmhop. All three backends can be enabled side by side and mixed freely in the same configuration.
llama.cpp runs as a native, hardened systemd system unit under DynamicUser
.
sglang and vLLM are launched as rootless Podman containers through quadlet-nix.
Each Quadlet backend gets a dedicated, lingering system user (sglang
, vllm
) that owns its cache directory, sub-UID range and rootless container store.
The container units are installed under that user's per-UID search path and therefore run as systemd user units, not system units.
This is a deliberate workaround for NVIDIA/nvidia-container-toolkit#648:
nvidia-cdi-hook
runs as an OCI createContainer
hook inside the container's user namespace and fails to read the OCI bundle's config.json
whenever Podman uses a UID-mapped namespace (e.g., --userns auto
or --userns nomap
), which is the mode you end up in when systemd's system manager launches a rootless container.
Running each Quadlet unit under a real, lingering system user's systemd instance keeps Podman in the keep-id
-style mapping where the CDI hook can read the bundle and the GPU is correctly exposed. No worker ever runs as root.
For convenience, the module injects a tiny per-backend helper into environment.systemPackages
whenever the backend's default user is used:
llama-cpp
workers are plain system units, so they are managed with the usualsystemctl status llama-cpp-<model>
andjournalctl -u llama-cpp-<model>
.sglang-shell
andvllm-shell
arewriteShellApplication
wrappers aroundmachinectl shell
that drop you into the backend user's session, wheresystemctl --user
,journalctl --user
andpodman ps
see the worker units directly. Run them with no arguments for an interactive shell, or pass a command to execute it inside the session.
services.llmhop = {
enable = true;
llama-cpp = {
enable = true;
models."qwen3-8b" = {
port = 18001;
settings.hf-repo = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
};
};
sglang = {
enable = true;
models."qwen3-coder" = {
port = 19001;
model = "Qwen/Qwen3-8B";
settings.reasoning-parser = "qwen3";
};
};
vllm = {
enable = true;
models."llama-3-8b" = {
port = 20001;
model = "meta-llama/Meta-Llama-3-8B-Instruct";
};
};
};
See the options reference for the full list of per-backend options.
The generated config file lives in the world-readable Nix store, so secrets should never be placed in services.llmhop.settings
directly.
Instead, reference them via ${file:...}
and hand the files to the service with systemd's LoadCredential=
.
The right-hand side of each LoadCredential
entry is just a file path, so anything that produces a file works: agenix or sops-nix outputs, a manually-managed file under /etc/llmhop/
, or a path emitted by your own secret-provisioning tool.
services.llmhop.settings = {
authTokens = [ "\${file:client_token}" ];
models."openai-gpt-4o" = {
url = "https://api.openai.com";
headers.Authorization = "Bearer \${env:OPENAI_KEY}";
};
};
systemd.services.llmhop.serviceConfig = {
LoadCredential = [ "client_token:/etc/llmhop/client-token" ];
EnvironmentFile = [ "/etc/llmhop/openai.env" ];
};
/etc/llmhop/openai.env
is a plain KEY=VALUE
file:
OPENAI_KEY=sk-...
${file:...}
references are resolved against $CREDENTIALS_DIRECTORY
, which systemd exposes as a per-unit tmpfs accessible only to this service, compatible with DynamicUser
and the rest of the sandbox.
${env:...}
picks up anything the unit inherits, typically via EnvironmentFile=
. Pick whichever matches how your secret tooling hands you the data; mixing both in one config is fine.