Route to your own hardware — or the cloud, when you choose. Refuse anything over budget. Kill loops in seconds. One ~5.5 MB Rust binary that decides before the money is spent, not a dashboard that reports it after.
Works with any OpenAI-compatible agent — point base_url
at localhost:8787/v1
.
Claude Code connects via ANTHROPIC_BASE_URL
. Pre-release · MIT · dogfooded daily.
An agent that starts looping at 2 a.m. keeps spending until someone wakes up. The billing alert is a receipt, not a control — it fires after the damage. Dashboards chart what happened. Stoke decides what is allowed to happen.
A retrying agent will happily replay the same failing request all night, on your metered key.
Spend notifications trail actual usage. By the time one fires, the budget is already gone.
An over-budget or looping request never reaches the provider. Refused at the gateway, not flagged in a chart.
Limits that reject requests before a provider ever sees them — not alerts for a human to find in the morning.
Per-key spend caps in USD, set in a [[keys]]
config table (each key must also be listed in STOKE_API_KEYS
) and applied to the budget guard at startup. Checked before any provider call: over the cap, the request is refused with 429 Budget exceeded
— an error your agent can handle, not a log line for later. Live spend per key at /v1/budget
; per-request cost rides along in a stoke_cost
field on non-streaming responses. Streamed responses accrue spend too — Stoke reads the usage the provider reports as the bytes pass through, and charges when the stream ends or the client hangs up. A metered provider that reports nothing is billed from an estimate, flagged as one, never as $0.
A two-layer circuit breaker: exact prompt-hash matching plus opt-in semantic similarity, so a loop that rephrases itself still counts as a loop. Five similar requests within 60 seconds blocks the key for 120 seconds. Thresholds are global constants today — per-key tuning is on the roadmap.
A sliding 60-second window per API key, set as rate_limit_rpm
in the same [[keys]]
table and applied at startup. One misbehaving agent gets refused (429); everyone else keeps working.
No API keys configured and no dev flag: every request is rejected. Every endpoint except /health
requires auth — status endpoints included, because model inventory and node load are reconnaissance data. Federation polling authenticates too. Misconfigured means closed, not open.
Stoke is dogfooded daily on a real MacBook + Mac Studio + Ollama Cloud setup. Point it at your Ollama nodes and it routes like it knows them — because it polls them.
A background registry polls each node's /api/tags
(models pulled) and /api/ps
(models loaded in RAM). Placement prefers warm nodes, balances the same model across machines by live in-flight count, breaks ties with a latency EWMA, and fails over automatically. Unreachable nodes are excluded until they come back.
Every live stream teaches it: time-to-first-token and tokens/sec are measured per model per node, and context windows plus tool-calling support come from /api/show
— so predictions come from your hardware, not guesses.
Provider type "stoke"
lets one gateway front another. The peer's models, warm state, live load, and measured speeds are imported through its /v1/nodes
— richer state than raw Ollama can report. Forwarded requests carry x-stoke-hop
; depth is exactly one, and a deliberate A-to-B-to-A cycle cannot loop — a shipped test script proves it.
The practical win: Ollama stays bound to 127.0.0.1
. Stoke is the only network-facing door, and it authenticates.
Point your agent at the auto
model and Stoke scores every eligible (model, node) pair on estimated cost, predicted latency, and your stated preference order — then picks. auto-cheap
weights spend, auto-fast
weights speed. Models come only from your config and your discovered nodes; Stoke ships with zero model names.
Hard exclusions, not vibes: prompts that don't fit a model's context window and tool-calling requests aimed at models that can't emit structured tool_calls
are ruled out with the reason on the receipt.
Opt-in (hedge = true
): small prompts whose model sits warm on two of your machines are fired at both — first past prefill wins, the loser is dropped. Your idle hardware races for you; duplicate local compute buys tail latency that no single node can.
Zero-marginal nodes only — hedging never doubles a cloud bill.
Auto-routed responses carry the full receipt: the chosen candidate with estimated cost and predicted latency, every alternative and why it lost, and an honest counterfactual — the list price your configured cloud model would have charged. GET /v1/budget
aggregates them into a running ledger. Estimates, clearly labeled; never a quality claim.
"stoke_route": { "node": "mac-studio", "auto": {
"mode": "auto", "class": "Code",
"chosen": { "model": "your-35b", "node": "mac-studio", "warm": true, "est_cost_usd": 0.0, "predicted_ms": 2100 },
"not_taken": [{ "model": "your-8b", "node": "macbook", "warm": false, "est_cost_usd": 0.0, "predicted_ms": 9800 }],
"counterfactual_usd_est": 0.0214 // what your configured cloud model would have charged
}}
This is the actual order of operations for one request. Every stage before the provider call can refuse it — and four of them are yours to write.
The yours stages take webhooks in any language — a plain HTTP endpoint is a plugin. JS/TS plugins run on an embedded V8 behind a compile-time flag. Nothing is a black box: every stage above the provider call can say no, and the ones that transform your prompt run before it leaves the machine.
You run coding agents — Claude Code, OpenCode, aider, Codex — against metered API keys, and one bad loop is a four-figure morning. Stoke puts a loop kill switch, a rate limit, and a hard USD cap between the agent and the key. The loop kill switch and rate limit apply to every request, streaming included — so a runaway loop is stopped no matter how it talks. OpenCode works end-to-end today; any OpenAI-compatible agent connects via base_url
.
A MacBook here, a Mac Studio there, maybe a Linux box with a GPU. Stoke fronts them all: warm-aware placement, live load balancing of the same model across machines, automatic failover, and federation between gateways — while Ollama never gets exposed to the LAN.
You ship an AI product and need usage entitlement enforcement: hard per-tenant caps your billing team can trust, enforced at the request path — a control plane over every customer's LLM spend. This is where Stoke is headed, not shipped yet; we are looking for design partners.
If your agents run on a subscription plan — Claude Max or Pro, ChatGPT Plus — there is no per-request dollar price, so there is nothing to dollar-cap. Stoke can still rate-limit that traffic and kill loops in it, but it cannot cap it in dollars, and we won't pretend otherwise. Hard USD caps apply to metered API keys, where every request has a price.
Data residency by architecture, not by contract. An EU-based project, built for people who read data-processing agreements and would rather not need them.
Requests route to machines you own. Nothing goes to a cloud provider unless you explicitly configure one. Once configured, automatic failover may use it — so if prompts must never leave your machines, simply don't configure a cloud tier.
Ollama stays bound to 127.0.0.1
. Stoke is the only network-facing surface, and every endpoint except /health
requires auth.
Model inventory and node load are reconnaissance data, so even the status endpoints authenticate. No keys configured means no service — not open service.
LiteLLM, Portkey and Helicone are good tools. They cover far more providers and go much deeper on observability. Stoke is built for a different job: enforcement.
| Stoke | LiteLLM | Portkey | Helicone | |
|---|---|---|---|---|
| Primary job | Enforcement — refuse bad requests at the gateway | Universal gateway | Gateway + observability | Observability |
| Loop kill switch | Exact-hash + semantic circuit breaker | — | — | — |
| Fail-closed default | No keys configured = all requests rejected | Configuration-dependent | Configuration-dependent | Configuration-dependent |
| Node-aware local routing | Warm state + live load via /api/ps |
|||
| — | — | — | ||
| Gateway federation | Stoke-behind-Stoke, hop-guarded, cycle-safe | — | — | — |
| Deployment | One ~5.5 MB Rust binary, zero runtime deps | Python service (database needed for key management features) | Hosted, or self-host stack | Hosted, or self-host stack |
| Provider coverage | OpenAI-compatible + Ollama + federation. They win here. | 100+ providers | Broad | Broad |
| Observability depth | Per-response route + cost, audit log. Deliberately minimal — they win here. | Deep | Deep dashboards | Their core strength |
No benchmark numbers in this table, on purpose. When we publish performance claims, they will ship with a reproducible harness first.
Static binaries for macOS and Linux are built from every commit on main
and published to a rolling nightly
build — no Rust toolchain needed. Honest status: that rolling build is what exists today; the first tagged stable release is still to come, and the installer falls back to compiling from source on any platform we don't ship.
curl -sSf https://stokegate.com/install | sh -s -- --version nightly
cp stoke.example.toml stoke.toml # your nodes, your providers
STOKE_API_KEYS=team-key stoke
Prefer to build it yourself? git clone
and cargo build --release
— the installer does exactly that when no binary matches your platform.
A minimal stoke.toml
— one local Ollama node, default port 8787:
[server]
host = "127.0.0.1"
port = 8787
[[providers]]
name = "mac-studio"
type = "openai_compatible"
base_url = "http://127.0.0.1:11434/v1"
tier = "local"
Then point any agent at Stoke instead of the provider — OpenAI-compatible tools and Claude Code both work:
OPENAI_BASE_URL=http://localhost:8787/v1 # OpenCode, aider, Codex
ANTHROPIC_BASE_URL=http://localhost:8787 # Claude Code (/v1/messages)
Note the fail-closed default: without STOKE_API_KEYS
(or STOKE_DEV=1
for local experiments), Stoke rejects every request. That is a feature.
Stoke ships features when they earn their place — dogfooded first, not promised against a roadmap. What gets built next is driven by what real users run into. If Stoke is one capability away from working for you, that's the most useful thing you can tell us.
Open questions we're weighing — weigh in if one is yours:
Today failover retries the same model on another machine. Should a role fall through a chain — cloud model hits a 429 or an outage, traffic degrades to your local model, even across vendors? Tell us how you'd want it to behave.
Hard usage-entitlement enforcement per customer, at the request path. If your pricing tiers need teeth, start a thread.
Enforcement benchmarks, OpenTelemetry export, Anthropic↔local translation — all on the table. Tell us which one unblocks you and it moves up.
The repo ships reproducible test harnesses. Run them on your own machine and watch the claims hold or break.
One gateway against two mock Ollama nodes — no Ollama install needed: model discovery, warm placement, streaming in-flight accounting, failover, health.
Two gateways wired into a deliberate cycle: inventory import, cross-gateway routing, hop guard, cycle safety.
While they run, poll GET /v1/nodes
— the live registry every routing decision is made from.
Enforcement benchmarks — overhead, time-to-trip, false-positive rate — ship with a public reproducible harness. Numbers only when you can run them yourself.
Open source, MIT licensed, no signup, no lock-in. Pre-release and built in the open.