Self-hosted LLM gateway. Cost effective, deterministic, fast. Secure and private by default.
Foreman is a self-hosted LLM gateway: one Go binary between your coding agent and any number of providers. Your keys and traffic stay inside your network. Track every dollar of spend, and switch models or providers from inside your favorite harness with no code changes. Keep frontier models where quality matters and route routine work to cheaper ones. You write the policy, so you cut cost without giving up output quality.
It sends each request to the cheapest model your policy allows, and pins every conversation to its model while the provider's prompt cache is warm, so routing never burns your cache to save pennies. OpenAI and Anthropic surfaces in, any number of providers out, and every decision lands in a ledger you can query for the reason it made and the alternatives it rejected.
Install with Homebrew (macOS):
brew install northwood-systems/tap/foreman
On Linux or from source: go install github.com/Northwood-Systems/foreman/cmd/foreman@latest
, a prebuilt archive from Releases, or go build -o bin/foreman ./cmd/foreman
(Go 1.26+).
Then run the interactive setup to pick providers, assign cheap/mid/frontier models, and generate a Foreman key (shown once, stored only as a hash):
foreman init # never asks for a secret: provider keys stay in your env vars,
foreman serve # referenced as ${VAR} and resolved at startup
Point your coding agent at it. Claude Code, Codex, OpenCode, and Pi setups are in docs/clients.md. Every response carries X-Foreman-Trace-ID
, X-Foreman-Model
, X-Foreman-Provider
, and X-Foreman-Route-Reason
headers; foreman trace <id>
shows the full decision, including every alternative the router rejected and why.
Pick the tier by precedence: policy floors (high risk and planning always get frontier) > explicitX-Foreman-Task-Type
/X-Foreman-Risk
headers (the precision instrument for harnesses you control) > the model the client asked for, since your agent's native model picker is the tier switch. Foreman advertisesforeman-plan
/foreman-code
/foreman-fast
via/v1/models
, and also resolves catalog names and upstream model names (a request forclaude-sonnet-4-5
lands in whatever tier your catalog gives that model). Unrecognized models fall back to the documented defaults (routine_code
/low
). Message content is never inspected.Route within the tier: the router picks the cheapest healthy model the tier allows; a tier is a list, so multiple models across providers compete on price and cover for each other.Stick while warm: withX-Foreman-Session-ID
, a conversation keeps its model/provider while the prompt cache is warm (TTL from the catalog'sprompt_cache.ttl_seconds
). Explicit signals (switching the model picker, or a task-type header) release a warm route across tiers; unlabeled traffic never thrashes the cache.Fail over: provider errors trip a circuit breaker (threshold + cooldown + half-open trials) and requests retry on the next allowed route.Record: every request writes a ledger event (measured tokens, measured cost) and a route trace (reason + rejected alternatives). SQLite by default, Postgres for production.
Passthrough is faithful: tools, images, and parameters Foreman doesn't model are forwarded byte-identical to same-protocol upstreams. For cross-protocol traffic (Anthropic-format clients on OpenAI-compatible providers), the full tool protocol is translated: tool definitions, tool results, and streamed tool calls, so Claude Code can run against Qwen/GLM-class upstreams. Only what cannot translate (images, documents) is refused with an explicit 400, never silently flattened.
foreman spend # totals, cache read rate, per-team breakdown
foreman models # model/provider mix
foreman trace <id> # one routing decision, fully explained
foreman top # live view (bubbletea)
Everything the CLI shows comes from GET /v1/foreman/*
JSON APIs (spend/summary
, spend/top
, models/mix
, cache/metrics
, traces/{id}
, ledger/export
), so your own dashboards can read the same numbers. --json
on any subcommand pipes cleanly.
Foreman never publishes a savings number. It is the instrument that computes yours, from your own ledger: actual_cost_usd
is measured usage priced by your catalog (or provider-reported cost), never a backfilled estimate. Send your own traffic, read your own receipt.
Two JSON files and environment variables, documented in docs/policy.md:
configs/policy.example.json
: API keys (SHA-256 hashes), and model roles: which models arecheap
/mid
/frontier
and which task types each role is allowed to handle.configs/models.example.json
: the model catalog, with provider, upstream model name, prices per MTok (fresh/cached/output/cache-write), and prompt-cache TTL.- Provider credentials via env only:
FOREMAN_OPENAI_COMPAT_PROVIDERS=qwen,openai
plusFOREMAN_PROVIDER_<ID>_BASE_URL
/_API_KEY
,FOREMAN_ANTHROPIC_API_KEY
for native Anthropic, andFOREMAN_BEDROCK_REGION
- AWS credentials for Claude via AWS Bedrock. Knobs:
FOREMAN_WARM_ROUTE_TTL
,FOREMAN_PROVIDER_FAILURE_THRESHOLD
,FOREMAN_PROVIDER_FAILURE_COOLDOWN
,FOREMAN_REQUEST_TIMEOUT
,FOREMAN_STREAM_TIMEOUT
,FOREMAN_MAX_BODY_BYTES
,FOREMAN_CONCURRENCY_LIMIT
(docs/gateway-api.mdhas the full list).
Logs are structured JSON with request/trace IDs, route metadata, and team attribution; API keys and prompt payloads are never logged.
Breaking large jobs into small, verifiable subtasks that cheap models execute, escalating only the pieces that fail deterministic verification, is something you build on top of the gateway, in whatever language your harness lives in. The whole pattern is four HTTP calls per subtask: plan with X-Foreman-Task-Type: planning
(frontier per policy), execute each piece with a cheap-class task type, verify the output deterministically yourself, and re-send failures with X-Foreman-Task-Type: escalation
. Every call goes through the gateway, so policy and spend accounting apply to every subtask, and /v1/foreman/spend/summary
gives you the running cost to enforce a budget against. See docs/design.md.
Design · Gateway API · Policy & Catalog · Client Setup · Deployment
Apache-2.0