Show HN: Relay – a self-hosted LLM gateway with eval-gated routing

Self-hosted LLM gateway + model router. One static Go binary that speaks both the OpenAI and Anthropic API dialects inbound, routes across any provider outbound (bring your own keys), with routing policies from static aliases to a learned smart tier — gated by an eval harness that ships in the box. Zero telemetry — not even opt-in pings. Apache-2.0.

any OpenAI SDK ─┐                                  ┌─ OpenAI / Anthropic / Gemini
Claude Code ────┤→  relay (one binary, :4000)  →  ├─ Groq / DeepSeek / Mistral / xAI / …
plain curl ─────┘   route · failover · log · cache └─ your local Ollama
brew install llmrelay/tap/relay

export GEMINI_API_KEY=...        # and/or OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, ...
relay serve

curl http://localhost:4000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"gemini/gemini-3.1-flash-lite","messages":[{"role":"user","content":"hello"}]}'
export ANTHROPIC_BASE_URL=http://localhost:4000   # Claude Code now runs through relay

relay init

scaffolds a full relay.yaml — including where to get every provider's key. See examples/ for Claude Code, Cursor, SDK snippets, and the sensitive-local / bulk-cheap alias recipe.

relay | LiteLLM | OpenRouter | RouteLLM | | |---|---|---|---|---| | What it is | self-hosted gateway + router | Python proxy/SDK gateway | hosted aggregator API | research routing framework | | Deploy | one static binary / distroless image | Python service + deps | nothing (their cloud) | Python library | | Inbound dialects | OpenAI and Anthropic, full cross-translation | OpenAI (+ passthroughs) | OpenAI-compatible | n/a | | Learned routing | tiered, local, trained on your logs, eval-gated | manual routing strategies | their routing | the prior art¹ | | Eval harness in the box | yes (relay eval , committed sets, live-judge) | no | no | offline research harness | | Your prompts transit a third party | never (self-hosted; remote-embedder routing requires explicit opt-in) | self-hosted | yes — that's the product | n/a | | Telemetry | none, ever | none per their docs | hosted service | n/a | | Ecosystem breadth | 4 native + 12 presets, adapter guide | broadest provider matrix, teams/budgets/virtual keys | very broad | n/a | | License / runtime | Apache-2.0, Go | mixed (OSS + enterprise), Python | proprietary service | Apache-2.0, Python |

¹ relay's tier-2 KNN is the same family as RouteLLM (Ong et al.), run locally over your own traffic; graph routers (GraphRouter, ICLR 2025) are roadmap. LiteLLM and OpenRouter are good products with different trade-offs — pick the row that matters to you.

Gateway overhead measured against a loopback mock upstream (go test -run TestOverheadBudget ./internal/server/

— the budget is a hard CI gate, methodology in DESIGN.md §11; Windows 11 / Go 1.25, 2026-07-18):

Provider latency dominates end-to-end time; relay's job is to stay invisible.

Static routes and aliases with four policies (fallback / cheapest / fastest / weighted), plus reliability underneath every chain: retries with jittered backoff, API-key pools with rate-limit cooldowns, circuit breakers, and pre-first-token streaming failover.

relay ships a difficulty-based smart router (easy traffic → your cheap chain, hard → your frontier chain) with an eval harness — and the harness's own verdict is that you should beat our baseline on your traffic before trusting it, so smart routing is off by default:

relay eval                          # dry-run: the committed sets, your candidates
relay eval --live-judge --dry-run   # real completions, judge-scored; prints spend first

What our harness measured on the held-out set v2, live-judged — real completions from each routed model, quality scored by claude-opus-4-8

(2026-07-20; 49 prompts, valid N = 49 for every policy; completions capped at 700 output tokens; mid band is claude-sonnet-5

because Gemini's free tier caps gemini-3.5-flash

at 20 requests/day; 6 of 147 judge replies buried the verdict number in commentary and were re-parsed deterministically from the preserved raw replies rather than scored 0 — the corrections log inside the verdict JSON records each one):

Values are shown rounded. Gate decisions use the unrounded values from the committed verdict asset (a delta of exactly −0.0200 would pass — the rule is inclusive; tier 1's unrounded delta is −0.020408…, exactly −1/49, strictly below it). A CI test asserts this table matches the asset at full precision.

On this 49-prompt, 700-output-token, single-judge evaluation, the static-cheap baseline received a higher judged score than the configured frontier baseline. That result is specific to the tested models, prompts, token cap, and judge — it is not a general model-quality claim. What it does establish is the bar relay holds itself to: routing cleverness must earn its keep against strong dumb baselines, measured on your traffic. Tier 1 failed that bar on held-out data; tier 2 passed within tolerance but stays opt-in for v0.1.0; no smart tier is on by default.

(History: on the v1 set tier 1 had "passed" synthetic at −0.017 — in-sample flattery, since its weights were calibrated on v1. On v2 synthetic labels it failed at −0.071. v1 is the calibration set; v2 live-judged is the standing verdict. Full tables: assets/eval/

.)

Enabling it is one config block — with an explicit tier, because nothing routes your traffic by silent default:

providers:
  gemini:
    api_key: ["${GEMINI_API_KEY}"]
  anthropic:
    api_key: ["${ANTHROPIC_API_KEY}"]
  ollama:
    type: ollama

routing:
  default: smart
  smart:
    easy: gemini/gemini-3.1-flash-lite
    hard: anthropic/claude-sonnet-5
    embeddings: ollama/nomic-embed-text   # selects the knn tier (documented path)

Tier 2 (KNN) gets better on YOUR traffic via relay train — implicit signals, optional replay+judge (always estimates spend and asks first), and

POST /v1/feedback

scores grow its reference set; then measure your own crossover with relay eval --refs ~/.relay/smart_refs.json

. Every smart decision logs its evidence (knn: 5 neighbors [seed-math-03 d=0.75 sim=0.95; …]

) — "the model felt like it" is not an accepted routing reason. A remote embedder requires allow_remote_embeddings: true

; routing never silently ships prompts anywhere.

relay compare --models gemini/gemini-3.1-flash-lite,anthropic/claude-haiku-4-5,groq/llama-3.3-70b \
  --html compare.html "Explain CRDTs to a backend engineer in five sentences."

One table (and a shareable HTML report): output, cost, latency, and TTFT side by side, through the same adapters that serve your traffic. docs/models-landscape.md is the dated "which model for what" companion.

Every request logs to local SQLite with privacy tiers (off

by default — metadata only; embeddings

stores query vectors, never text; full

is an explicit choice). /dashboard

shows spend by day, latency percentiles, and recent routing decisions with human-readable reasons. /metrics

is Prometheus. Models missing from the pricing registry surface as unpriced — never a silent $0.

Loopback by default; a non-loopback bind without inbound API keys refuses to start. No telemetry. See SECURITY.md.

CONTRIBUTING.md — the adapter-authoring guide is there; an OpenAI-compatible provider is a one-entry preset. Design changes get argued in DESIGN.md §0 first; it's how the doc stays true.