Show HN: Foreman, a self-hosted LLM gateway for cost aware model routing

Foreman, a self-hosted LLM gateway, routes requests to the cheapest model allowed by user-defined policies while maintaining deterministic behavior and prompt cache efficiency. The open-source tool supports multiple providers, tracks spending, and requires no code changes to switch models, aiming to reduce costs without sacrificing output quality.

Self-hosted LLM gateway. Cost effective, deterministic, fast. Secure and private by default. Foreman is a self-hosted LLM gateway: one Go binary between your coding agent and any number of providers. Your keys and traffic stay inside your network. Track every dollar of spend, and switch models or providers from inside your favorite harness with no code changes. Keep frontier models where quality matters and route routine work to cheaper ones. You write the policy, so you cut cost without giving up output quality. It sends each request to the cheapest model your policy allows, and pins every conversation to its model while the provider's prompt cache is warm, so routing never burns your cache to save pennies. OpenAI and Anthropic surfaces in, any number of providers out, and every decision lands in a ledger you can query for the reason it made and the alternatives it rejected. Install with Homebrew macOS : brew install northwood-systems/tap/foreman On Linux or from source: go install github.com/Northwood-Systems/foreman/cmd/foreman@latest , a prebuilt archive from Releases https://github.com/Northwood-Systems/foreman/releases , or go build -o bin/foreman ./cmd/foreman Go 1.26+ . Then run the interactive setup to pick providers, assign cheap/mid/frontier models, and generate a Foreman key shown once, stored only as a hash : foreman init never asks for a secret: provider keys stay in your env vars, foreman serve referenced as ${VAR} and resolved at startup Point your coding agent at it. Claude Code, Codex, OpenCode, and Pi setups are in docs/clients.md /Northwood-Systems/foreman/blob/main/docs/clients.md . Every response carries X-Foreman-Trace-ID , X-Foreman-Model , X-Foreman-Provider , and X-Foreman-Route-Reason headers; foreman trace <id shows the full decision, including every alternative the router rejected and why. Pick the tier by precedence: policy floors high risk and planning always get frontier explicit X-Foreman-Task-Type / X-Foreman-Risk headers the precision instrument for harnesses you control the model the client asked for, since your agent's native model picker is the tier switch. Foreman advertises foreman-plan / foreman-code / foreman-fast via /v1/models , and also resolves catalog names and upstream model names a request for claude-sonnet-4-5 lands in whatever tier your catalog gives that model . Unrecognized models fall back to the documented defaults routine code / low . Message content is never inspected. Route within the tier: the router picks the cheapest healthy model the tier allows; a tier is a list, so multiple models across providers compete on price and cover for each other. Stick while warm: with X-Foreman-Session-ID , a conversation keeps its model/provider while the prompt cache is warm TTL from the catalog's prompt cache.ttl seconds . Explicit signals switching the model picker, or a task-type header release a warm route across tiers; unlabeled traffic never thrashes the cache. Fail over: provider errors trip a circuit breaker threshold + cooldown + half-open trials and requests retry on the next allowed route. Record: every request writes a ledger event measured tokens, measured cost and a route trace reason + rejected alternatives . SQLite by default, Postgres for production. Passthrough is faithful: tools, images, and parameters Foreman doesn't model are forwarded byte-identical to same-protocol upstreams. For cross-protocol traffic Anthropic-format clients on OpenAI-compatible providers , the full tool protocol is translated: tool definitions, tool results, and streamed tool calls, so Claude Code can run against Qwen/GLM-class upstreams. Only what cannot translate images, documents is refused with an explicit 400, never silently flattened. foreman spend totals, cache read rate, per-team breakdown foreman models model/provider mix foreman trace <id one routing decision, fully explained foreman top live view bubbletea Everything the CLI shows comes from GET /v1/foreman/ JSON APIs spend/summary , spend/top , models/mix , cache/metrics , traces/{id} , ledger/export , so your own dashboards can read the same numbers. --json on any subcommand pipes cleanly. Foreman never publishes a savings number. It is the instrument that computes yours, from your own ledger: actual cost usd is measured usage priced by your catalog or provider-reported cost , never a backfilled estimate. Send your own traffic, read your own receipt. Two JSON files and environment variables, documented in docs/policy.md /Northwood-Systems/foreman/blob/main/docs/policy.md : configs/policy.example.json : API keys SHA-256 hashes , and model roles: which models are cheap / mid / frontier and which task types each role is allowed to handle. configs/models.example.json : the model catalog, with provider, upstream model name, prices per MTok fresh/cached/output/cache-write , and prompt-cache TTL.- Provider credentials via env only: FOREMAN OPENAI COMPAT PROVIDERS=qwen,openai plus FOREMAN PROVIDER <ID BASE URL / API KEY , FOREMAN ANTHROPIC API KEY for native Anthropic, and FOREMAN BEDROCK REGION + AWS credentials for Claude via AWS Bedrock. Knobs: FOREMAN WARM ROUTE TTL , FOREMAN PROVIDER FAILURE THRESHOLD , FOREMAN PROVIDER FAILURE COOLDOWN , FOREMAN REQUEST TIMEOUT , FOREMAN STREAM TIMEOUT , FOREMAN MAX BODY BYTES , FOREMAN CONCURRENCY LIMIT docs/gateway-api.md /Northwood-Systems/foreman/blob/main/docs/gateway-api.md has the full list . Logs are structured JSON with request/trace IDs, route metadata, and team attribution; API keys and prompt payloads are never logged. Breaking large jobs into small, verifiable subtasks that cheap models execute, escalating only the pieces that fail deterministic verification, is something you build on top of the gateway, in whatever language your harness lives in. The whole pattern is four HTTP calls per subtask: plan with X-Foreman-Task-Type: planning frontier per policy , execute each piece with a cheap-class task type, verify the output deterministically yourself, and re-send failures with X-Foreman-Task-Type: escalation . Every call goes through the gateway, so policy and spend accounting apply to every subtask, and /v1/foreman/spend/summary gives you the running cost to enforce a budget against. See docs/design.md /Northwood-Systems/foreman/blob/main/docs/design.md . Design /Northwood-Systems/foreman/blob/main/docs/design.md · Gateway API /Northwood-Systems/foreman/blob/main/docs/gateway-api.md · Policy & Catalog /Northwood-Systems/foreman/blob/main/docs/policy.md · Client Setup /Northwood-Systems/foreman/blob/main/docs/clients.md · Deployment /Northwood-Systems/foreman/blob/main/docs/deployment.md Apache-2.0