# Keystone 1.0: what stable means for a self-hosted LLM gateway

> Source: <https://blog.ai-process.dev/2026/10/27/keystone-1-0/>
> Published: 2026-10-27 07:00:00+00:00

**Key takeaway:** Keystone 1.0 lets services request an effort level (LOW/MEDIUM/HIGH) plus attributes instead of a hardcoded model name, with prioritized rules editable at runtime mapping to fallback chains, so swapping in a new model is a rule edit rather than a change across every service.

# Keystone 1.0: what stable means for a self-hosted LLM gateway

Keystone. The Model Switchboard. (KTMS) reaches 1.0 together with ktms-admin and light-tokenmeter. What the gateway does, what is now stable across 1.x, how we test that promise, and what changed on the way.

**A model name in your code goes stale the day the next model ships.** With ten services, you
rewrite it ten times, and spread vendor keys across ten configs.

**Keystone. The Model Switchboard. (KTMS)** takes that out of your code. A service says how hard
the task is; the gateway decides which model to call, with which key, and whether your own GPU
should take it. It idles at 7 MiB of memory, and from 1.0 on its API doesn’t break within 1.x.

The web console **ktms-admin** and the usage meter **light-tokenmeter** reach 1.0 with it. All three
are written in Go and licensed under Apache 2.0. The features below are as of the current 1.3.

## What Keystone does

Keystone sits between your services and your model vendors. Your services call one endpoint, and KTMS:

- keeps the vendor API keys, encrypted in Postgres with AES-256-GCM, so clients never see them;
- picks the model from rules, not from a name hardcoded in your code;
- falls back to the next model when a vendor fails or throttles;
- enforces requests per minute, daily token budgets and hard spend caps per key, user and team;
- sends one usage event per call to a sink such as light-tokenmeter.

### Effort instead of model names

A service doesn’t ask for `gpt-4o`. It asks for an effort, `LOW`, `MEDIUM` or `HIGH`, and adds
free-form attributes such as `language` or `task`. An ordered list of rules maps that to a chain of
models:

```
{"rules":[
  {"name":"spanish-high", "priority":30, "effort":"EFFORT_HIGH",
   "attributes":{"language":"es"}, "use":["gemini-2.5-pro","gpt-4o"], "enabled":true},
  {"name":"low-default", "priority":20, "effort":"EFFORT_LOW",
   "use":["deepseek-chat","gpt-4o-mini"], "enabled":true},
  {"name":"anything-else", "priority":1, "use":["gpt-4o-mini"], "enabled":true}
]}
```

Rules change at runtime over the admin API or the console. When a new model comes out, you edit a rule, not forty services.

### Two ways in

Typed gRPC (`ktms.v1`) for services, and an OpenAI-compatible HTTP API for everything that already
speaks OpenAI:

``` python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="ktms_<your key>")
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Say hi in three words."}],
)
print(reply.choices[0].message.content, reply.model)
```

Response headers (`X-KTMS-Vendor`, `X-KTMS-Rule`, `X-KTMS-Placement`) say who answered and why.

### What it does (as of 1.3)

- **Routing:** effort LOW/MEDIUM/HIGH plus attributes, prioritized rules changed at runtime, fallback chains,`model: "auto"` with a judge (250 ms, falls back to MEDIUM), canaries sticky per user, shadow traffic with evaluation, a dry run with`ktms route explain` .
- **Reliability:** circuit breaker, a model paused on`Retry-After` , one attempt is one vendor call, defined behaviour when Postgres or Redis fail.
- **Money and limits:** RPM/TPM and a model allowlist per key, daily and monthly spend caps per key and user, a monthly budget per team, prompt-cache pricing, vendor Batch APIs.
- **Your own GPUs:** a node agent for Ollama, vLLM, TGI and llama.cpp, admission by real queue depth, “own nodes only” placement, capacity levels and load schedules, a local usage log with`ktms-node stats` .
- **Security:** vendor keys in Postgres under AES-256-GCM with rotation, PII masking, prompt-injection detection, latent guardrails and a semantic cache (experimental).
- **API:** typed gRPC`ktms.v1` and an OpenAI-compatible HTTP API, streaming, tool calling, lossless JSON Schema, embeddings, speech, images, Judge.
- **Management:** the`ktms` CLI with contexts and`ktms init` , configuration as YAML (`plan` /`apply` ), an MCP server (`ktms mcp` ) for AI assistants, a Claude Code plugin, an`admin:read` scope, and the ktms-admin console.
- **Operations:**`ktms_*` metrics with suggested alerts, a Helm chart on OCI, signed images with SBOM and SLSA provenance, an upgrade test on every pull request.

## Footprint

Keystone is one Go process with no interpreter. On our stand of eleven self-hosted gateways
(Google Cloud `c3-standard-4`, 4 vCPU, mock upstream, two runs), it used about 7 MiB idle and
20 MiB at 200 RPS, added about 0.64 ms at p50, sustained 4,000 RPS, and answered its first request
about 170 ms after `docker start`. agentgateway, APISIX and Kong add less latency; only agentgateway
matches the memory. The full write-up is in [Go vs Python for an LLM gateway](https://blog.ai-process.dev/2026/10/26/go-vs-python-llm-gateway/).

## What changed on the way to 1.0

- **Organizations and teams.** Team keys share a model allowlist and a monthly budget that the
gateway enforces across all of them.
- **Vendor calls you can reason about.** Vendor SDK retries are off, so one router attempt is one
vendor call.`Retry-After` pauses a model for that vendor credential, and the chain moves on.
- **Defined behaviour when a dependency fails.** With Postgres down, keys verified in the last five
minutes keep working and routing serves from the last config snapshot. With Redis down, rate
limits fail open, and you choose what spend caps do. Usage events queue and are delivered once.
- **Lossless JSON Schema.**`enum` ,`format` ,`nullable` and`additionalProperties` reach the vendor.
Where a vendor can’t enforce a schema, Keystone checks the reply, retries once and falls back.
- **Local nodes by real queue depth.** The node agent reports the engine’s own queue (vLLM, TGI,
llama.cpp), and admission skips a node where the request would wait too long.
- **Safe key rotation.**`RotateEncryption` re-encrypts vendor keys and node tokens in one locking
transaction, and the full rotation runs against Postgres in CI.
- **Signed releases.** Images and`checksums.txt` are signed with keyless cosign; images carry an
SBOM and SLSA provenance;`govulncheck` runs on every pull request.
- **Apache 2.0** instead of MIT, which adds an explicit patent grant.

## The neighbours

**ktms-admin** manages models, rules, keys and vendor credentials in the browser, and has a
playground for trying a prompt through your rules. Sign-in is a password, Google or any OpenID
Connect provider, GitHub, or the access proxy you already run, limited to an allowlist.

**light-tokenmeter** answers “who is spending our tokens, and are they worth it?”: one event per
call in, summaries per project, user, action, vendor and model out. The console labels each group
greedy or worthy from those summaries.

## Try it

```
git clone https://github.com/ai-process/ktms.git && cd ktms
OPENAI_API_KEY=sk-... docker compose up -d --build
curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer ktms_dev_default_key" -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Say hi in three words."}]}'
```

Upgrading from 0.14.x is one step. From older 0.x releases, upgrade to 0.14.x first.

## What 1.0 promises

0.x made no compatibility promise. From 1.0 on, nothing below changes incompatibly within 1.x:

| Surface | What stays compatible | 
|---|---|
| gRPC `ktms.v1` | No RPC, field, enum value or message is removed or renamed; field numbers and types stay. | 
| HTTP `/v1` and`X-KTMS-*` headers | Paths, request and response fields, header names. | 
| Environment variables | Names, defaults and meaning. | 
| Database | Any 1.x release migrates the database of an earlier one by itself. No manual SQL. | 
| Metrics, usage events, node agent | `ktms_*` names and labels,`usage.v1` fields, agent and gateway compatible both ways. | 
| Error contract | Status codes and reason strings keep their meaning. | 

Some features work and are supported but may still change in a minor release, and say so with an
`Experimental:` comment in the proto: the judge’s effort choice for `model: "auto"`, shadow traffic
and its evaluation, the semantic cache and latent guardrails, `JudgeStream` and
`EvaluateRollouts`, and the YAML configuration format and the `ktms` command until the format reaches `v1`.

### How the promise is checked

A promise nobody tests is a wish. Every pull request and every release runs two gates:

- **`buf breaking`** compares the proto with the previous release.
- **An upgrade test** starts the previous release and the oldest release of the same major, lets
them write a database, migrates it with the new build, checks that the new build serves it, and
then runs the old release again on the migrated database. That last step is what makes rolling
upgrades safe.

The release workflow publishes binaries only after both pass, and Docker images only after the binaries.

## Next

Routing sensitive data to your own nodes only, hedged requests, a corrective retry for JSON, and a Grafana dashboard. None of it breaks the 1.x API.

- Code: [github.com/ai-process/ktms](https://github.com/ai-process/ktms)
- Docs: [ai-process.dev](https://ai-process.dev)
- Console: [github.com/ai-process/ktms-admin](https://github.com/ai-process/ktms-admin)
- Usage meter: [github.com/ai-process/light-tokenmeter](https://github.com/ai-process/light-tokenmeter)
