{"slug": "show-hn-relay-a-self-hosted-llm-gateway-with-eval-gated-routing", "title": "Show HN: Relay – a self-hosted LLM gateway with eval-gated routing", "summary": "Relay, a self-hosted LLM gateway and model router released as a single static Go binary under Apache-2.0, supports OpenAI and Anthropic API dialects inbound and routes across any provider outbound with eval-gated routing policies. The project claims zero telemetry, sub-millisecond overhead, and includes a built-in eval harness, positioning itself as a privacy-focused alternative to LiteLLM, OpenRouter, and RouteLLM.", "body_md": "Self-hosted LLM gateway + model router. One static Go binary that speaks both\nthe **OpenAI and Anthropic API dialects inbound**, routes across any provider\noutbound (bring your own keys), with routing policies from static aliases to a\nlearned smart tier — gated by an eval harness that ships in the box. Zero\ntelemetry — not even opt-in pings. Apache-2.0.\n\n```\nany OpenAI SDK ─┐                                  ┌─ OpenAI / Anthropic / Gemini\nClaude Code ────┤→  relay (one binary, :4000)  →  ├─ Groq / DeepSeek / Mistral / xAI / …\nplain curl ─────┘   route · failover · log · cache └─ your local Ollama\n# 1. install (or download a release binary; a tested Dockerfile is included to build yourself)\nbrew install llmrelay/tap/relay\n\n# 2. keys you already have, zero config\nexport GEMINI_API_KEY=...        # and/or OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, ...\nrelay serve\n\n# 3. point anything at it\ncurl http://localhost:4000/v1/chat/completions -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"gemini/gemini-3.1-flash-lite\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}'\nexport ANTHROPIC_BASE_URL=http://localhost:4000   # Claude Code now runs through relay\n```\n\n`relay init`\n\nscaffolds a full relay.yaml — including where to get every\nprovider's key. See [examples/](/llmrelay/relay/blob/main/examples) for Claude Code, Cursor, SDK\nsnippets, and the sensitive-local / bulk-cheap alias recipe.\n\nrelay |\nLiteLLM | OpenRouter | RouteLLM | |\n|---|---|---|---|---|\n| What it is | self-hosted gateway + router | Python proxy/SDK gateway | hosted aggregator API | research routing framework |\n| Deploy | one static binary / distroless image | Python service + deps | nothing (their cloud) | Python library |\n| Inbound dialects | OpenAI and Anthropic, full cross-translation |\nOpenAI (+ passthroughs) | OpenAI-compatible | n/a |\n| Learned routing | tiered, local, trained on your logs, eval-gated |\nmanual routing strategies | their routing | the prior art¹ |\n| Eval harness in the box | yes (`relay eval` , committed sets, live-judge) |\nno | no | offline research harness |\n| Your prompts transit a third party | never (self-hosted; remote-embedder routing requires explicit opt-in) | self-hosted | yes — that's the product | n/a |\n| Telemetry | none, ever | none per their docs | hosted service | n/a |\n| Ecosystem breadth | 4 native + 12 presets, adapter guide | broadest provider matrix, teams/budgets/virtual keys | very broad | n/a |\n| License / runtime | Apache-2.0, Go | mixed (OSS + enterprise), Python | proprietary service | Apache-2.0, Python |\n\n¹ relay's tier-2 KNN is the same family as [RouteLLM](https://arxiv.org/abs/2406.18665)\n(Ong et al.), run locally over your own traffic; graph routers\n([GraphRouter](https://arxiv.org/abs/2410.03834), ICLR 2025) are roadmap.\nLiteLLM and OpenRouter are good products with different trade-offs — pick the\nrow that matters to you.\n\n| Inbound | Status |\n|---|---|\nOpenAI Chat Completions (`/v1/chat/completions` ) |\nfull, incl. streaming, tools, vision |\nOpenAI Responses API (`/v1/responses` ) |\nnot yet — tracked v1.1 fast-follow |\nAnthropic Messages (`/v1/messages` , `count_tokens` ) |\nfull, incl. streaming, tools |\nEmbeddings (`/v1/embeddings` ) |\nOpenAI dialect; providers without an embeddings API answer an honest 404 |\n`/v1/models` , `/metrics` (Prometheus), `/dashboard` , `/v1/feedback` |\nyes |\n\nGateway overhead measured against a loopback mock upstream (`go test -run TestOverheadBudget ./internal/server/`\n\n— the budget is a hard CI gate,\nmethodology in [DESIGN.md §11](/llmrelay/relay/blob/main/DESIGN.md); Windows 11 / Go 1.25, 2026-07-18):\n\n| Metric | Budget (CI-gated) | Measured |\n|---|---|---|\n| Non-streaming p50 overhead | < 5 ms | ~0.18 ms |\n| Added time-to-first-token p50 (streaming) | < 2 ms | ~0.63 ms |\n\nProvider latency dominates end-to-end time; relay's job is to stay invisible.\n\nStatic routes and aliases with four policies (fallback / cheapest / fastest / weighted), plus reliability underneath every chain: retries with jittered backoff, API-key pools with rate-limit cooldowns, circuit breakers, and pre-first-token streaming failover.\n\nrelay ships a difficulty-based smart router (easy traffic → your cheap chain,\nhard → your frontier chain) with an eval harness — and the harness's own\nverdict is that **you should beat our baseline on your traffic before trusting\nit**, so smart routing is off by default:\n\n```\nrelay eval                          # dry-run: the committed sets, your candidates\nrelay eval --live-judge --dry-run   # real completions, judge-scored; prints spend first\n```\n\nWhat our harness measured on the held-out set v2, **live-judged** — real\ncompletions from each routed model, quality scored by `claude-opus-4-8`\n\n(2026-07-20; 49 prompts, valid N = 49 for every policy; completions capped at\n700 output tokens; mid band is `claude-sonnet-5`\n\nbecause Gemini's free tier\ncaps `gemini-3.5-flash`\n\nat 20 requests/day; 6 of 147 judge replies buried the\nverdict number in commentary and were re-parsed deterministically from the\npreserved raw replies rather than scored 0 — the corrections log inside the\nverdict JSON records each one):\n\n| Policy | Cost vs always-frontier | Judged quality delta | Verdict at −0.0200 tolerance (inclusive) |\n|---|---|---|---|\n| static-cheap / cheapest | −98% | +0.0163 | passes |\n| static-mid | −69% | +0.0041 | passes |\n| tier 1 (lexical) | −50% | −0.0204 |\nfail — strictly below the bound |\n| tier 2 (embedding KNN, cold-start) | −17% | −0.0061 | passes within tolerance; opt-in for v0.1.0 |\n| static-frontier (baseline) | — | — | — |\n\nValues are shown rounded. Gate decisions use the unrounded values from the committed verdict asset (a delta of exactly −0.0200 would pass — the rule is inclusive; tier 1's unrounded delta is −0.020408…, exactly −1/49, strictly below it). A CI test asserts this table matches the asset at full precision.\n\nOn this 49-prompt, 700-output-token, single-judge evaluation, the static-cheap\nbaseline received a higher judged score than the configured frontier baseline.\nThat result is specific to the tested models, prompts, token cap, and judge —\nit is **not** a general model-quality claim. What it does establish is the bar\nrelay holds itself to: routing cleverness must earn its keep against strong\ndumb baselines, *measured on your traffic*. Tier 1 failed that bar on held-out\ndata; tier 2 passed within tolerance but stays opt-in for v0.1.0; no smart\ntier is on by default.\n\n(History: on the v1 set tier 1 had \"passed\" synthetic at −0.017 — in-sample\nflattery, since its weights were calibrated on v1. On v2 synthetic labels it\nfailed at −0.071. v1 is the calibration set; v2 live-judged is the standing\nverdict. Full tables: `assets/eval/`\n\n.)\n\nEnabling it is one config block — with an explicit tier, because nothing routes your traffic by silent default:\n\n```\nproviders:\n  gemini:\n    api_key: [\"${GEMINI_API_KEY}\"]\n  anthropic:\n    api_key: [\"${ANTHROPIC_API_KEY}\"]\n  ollama:\n    type: ollama\n\nrouting:\n  default: smart\n  smart:\n    easy: gemini/gemini-3.1-flash-lite\n    hard: anthropic/claude-sonnet-5\n    embeddings: ollama/nomic-embed-text   # selects the knn tier (documented path)\n    # tier: lexical                       # experimental: failed the held-out gate\n```\n\nTier 2 (KNN) **gets better on YOUR traffic via relay train** — implicit\nsignals, optional replay+judge (always estimates spend and asks first), and\n\n`POST /v1/feedback`\n\nscores grow its reference set; then measure your own\ncrossover with `relay eval --refs ~/.relay/smart_refs.json`\n\n. Every smart\ndecision logs its evidence (`knn: 5 neighbors [seed-math-03 d=0.75 sim=0.95; …]`\n\n) — \"the model felt like it\" is not an accepted routing reason. A remote\nembedder requires `allow_remote_embeddings: true`\n\n; routing never silently\nships prompts anywhere.\n\n```\nrelay compare --models gemini/gemini-3.1-flash-lite,anthropic/claude-haiku-4-5,groq/llama-3.3-70b \\\n  --html compare.html \"Explain CRDTs to a backend engineer in five sentences.\"\n```\n\nOne table (and a shareable HTML report): output, cost, latency, and TTFT side\nby side, through the same adapters that serve your traffic.\n[docs/models-landscape.md](/llmrelay/relay/blob/main/docs/models-landscape.md) is the dated\n\"which model for what\" companion.\n\nEvery request logs to local SQLite with privacy tiers (`off`\n\nby default —\nmetadata only; `embeddings`\n\nstores query vectors, never text; `full`\n\nis an\nexplicit choice). `/dashboard`\n\nshows spend by day, latency percentiles, and\nrecent routing decisions with human-readable reasons. `/metrics`\n\nis\nPrometheus. Models missing from the pricing registry surface as **unpriced**\n— never a silent $0.\n\nLoopback by default; a non-loopback bind without inbound API keys **refuses\nto start**. No telemetry. See [SECURITY.md](/llmrelay/relay/blob/main/SECURITY.md).\n\n[CONTRIBUTING.md](/llmrelay/relay/blob/main/CONTRIBUTING.md) — the adapter-authoring guide is there;\nan OpenAI-compatible provider is a one-entry preset. Design changes get\nargued in [DESIGN.md](/llmrelay/relay/blob/main/DESIGN.md) §0 first; it's how the doc stays true.", "url": "https://wpnews.pro/news/show-hn-relay-a-self-hosted-llm-gateway-with-eval-gated-routing", "canonical_source": "https://github.com/llmrelay/relay", "published_at": "2026-07-20 23:18:00+00:00", "updated_at": "2026-07-20 23:53:26.416354+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["Relay", "OpenAI", "Anthropic", "Gemini", "Groq", "DeepSeek", "Mistral", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/show-hn-relay-a-self-hosted-llm-gateway-with-eval-gated-routing", "markdown": "https://wpnews.pro/news/show-hn-relay-a-self-hosted-llm-gateway-with-eval-gated-routing.md", "text": "https://wpnews.pro/news/show-hn-relay-a-self-hosted-llm-gateway-with-eval-gated-routing.txt", "jsonld": "https://wpnews.pro/news/show-hn-relay-a-self-hosted-llm-gateway-with-eval-gated-routing.jsonld"}}