cd /news/ai-infrastructure/keystone-1-0-what-stable-means-for-a… · home › topics › ai-infrastructure › article
[ARTICLE · art-164018] src=blog.ai-process.dev ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Keystone 1.0: what stable means for a self-hosted LLM gateway

Keystone. The Model Switchboard. (KTMS) reached version 1.0 alongside its ktms-admin web console and light-tokenmeter usage meter, with the Go-based, Apache 2.0-licensed gateway now guaranteeing its API won't break within the 1.x line. Keystone routes service requests by effort level (LOW, MEDIUM or HIGH) plus attributes through prioritized, runtime-editable rules to model fallback chains, keeping vendor API keys encrypted in Postgres with AES-256-GCM, and idles at 7 MiB of memory. The current release, 1.3, adds a node agent for Ollama, vLLM, TGI and llama.cpp, an OpenAI-compatible HTTP API and typed gRPC ktms.v1, and an MCP server via `ktms mcp`.

key takeaway

Keystone 1.0 lets services request an effort level (LOW/MEDIUM/HIGH) plus attributes instead of a hardcoded model name, with prioritized rules editable at runtime mapping to fallback chains, so swapping in a new model is a rule edit rather than a change across every service.

read7 min views51 publishedOct 27, 2026

Keystone. The Model Switchboard. (KTMS) reaches 1.0 together with ktms-admin and light-tokenmeter. What the gateway does, what is now stable across 1.x, how we test that promise, and what changed on the way.

A model name in your code goes stale the day the next model ships. With ten services, you rewrite it ten times, and spread vendor keys across ten configs.

Keystone. The Model Switchboard. (KTMS) takes that out of your code. A service says how hard the task is; the gateway decides which model to call, with which key, and whether your own GPU should take it. It idles at 7 MiB of memory, and from 1.0 on its API doesn’t break within 1.x.

The web console ktms-admin and the usage meter light-tokenmeter reach 1.0 with it. All three are written in Go and licensed under Apache 2.0. The features below are as of the current 1.3.

What Keystone does #

Keystone sits between your services and your model vendors. Your services call one endpoint, and KTMS:

  • keeps the vendor API keys, encrypted in Postgres with AES-256-GCM, so clients never see them;
  • picks the model from rules, not from a name hardcoded in your code;
  • falls back to the next model when a vendor fails or throttles;
  • enforces requests per minute, daily token budgets and hard spend caps per key, user and team;
  • sends one usage event per call to a sink such as light-tokenmeter.

Effort instead of model names

A service doesn’t ask for gpt-4o. It asks for an effort, LOW, MEDIUM or HIGH, and adds free-form attributes such as language or task. An ordered list of rules maps that to a chain of models:

{"rules":[
  {"name":"spanish-high", "priority":30, "effort":"EFFORT_HIGH",
   "attributes":{"language":"es"}, "use":["gemini-2.5-pro","gpt-4o"], "enabled":true},
  {"name":"low-default", "priority":20, "effort":"EFFORT_LOW",
   "use":["deepseek-chat","gpt-4o-mini"], "enabled":true},
  {"name":"anything-else", "priority":1, "use":["gpt-4o-mini"], "enabled":true}
]}

Rules change at runtime over the admin API or the console. When a new model comes out, you edit a rule, not forty services.

Two ways in

Typed gRPC (ktms.v1) for services, and an OpenAI-compatible HTTP API for everything that already speaks OpenAI:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="ktms_<your key>")
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Say hi in three words."}],
)
print(reply.choices[0].message.content, reply.model)

Response headers (X-KTMS-Vendor, X-KTMS-Rule, X-KTMS-Placement) say who answered and why.

What it does (as of 1.3)

  • Routing: effort LOW/MEDIUM/HIGH plus attributes, prioritized rules changed at runtime, fallback chains,model: "auto" with a judge (250 ms, falls back to MEDIUM), canaries sticky per user, shadow traffic with evaluation, a dry run withktms route explain .
  • Reliability: circuit breaker, a model d onRetry-After , one attempt is one vendor call, defined behaviour when Postgres or Redis fail.
  • Money and limits: RPM/TPM and a model allowlist per key, daily and monthly spend caps per key and user, a monthly budget per team, prompt-cache pricing, vendor Batch APIs.
  • Your own GPUs: a node agent for Ollama, vLLM, TGI and llama.cpp, admission by real queue depth, “own nodes only” placement, capacity levels and load schedules, a local usage log withktms-node stats .
  • Security: vendor keys in Postgres under AES-256-GCM with rotation, PII masking, prompt-injection detection, latent guardrails and a semantic cache (experimental).
  • API: typed gRPCktms.v1 and an OpenAI-compatible HTTP API, streaming, tool calling, lossless JSON Schema, embeddings, speech, images, Judge.
  • Management: thektms CLI with contexts andktms init , configuration as YAML (plan /apply ), an MCP server (ktms mcp ) for AI assistants, a Claude Code plugin, anadmin:read scope, and the ktms-admin console.
  • Operations:ktms_* metrics with suggested alerts, a Helm chart on OCI, signed images with SBOM and SLSA provenance, an upgrade test on every pull request.

Footprint #

Keystone is one Go process with no interpreter. On our stand of eleven self-hosted gateways (Google Cloud c3-standard-4, 4 vCPU, mock upstream, two runs), it used about 7 MiB idle and 20 MiB at 200 RPS, added about 0.64 ms at p50, sustained 4,000 RPS, and answered its first request about 170 ms after docker start. agentgateway, APISIX and Kong add less latency; only agentgateway matches the memory. The full write-up is in Go vs Python for an LLM gateway.

What changed on the way to 1.0 #

  • Organizations and teams. Team keys share a model allowlist and a monthly budget that the gateway enforces across all of them.
  • Vendor calls you can reason about. Vendor SDK retries are off, so one router attempt is one vendor call.Retry-After s a model for that vendor credential, and the chain moves on.
  • Defined behaviour when a dependency fails. With Postgres down, keys verified in the last five minutes keep working and routing serves from the last config snapshot. With Redis down, rate limits fail open, and you choose what spend caps do. Usage events queue and are delivered once.
  • Lossless JSON Schema.enum ,format ,nullable andadditionalProperties reach the vendor. Where a vendor can’t enforce a schema, Keystone checks the reply, retries once and falls back.
  • Local nodes by real queue depth. The node agent reports the engine’s own queue (vLLM, TGI, llama.cpp), and admission skips a node where the request would wait too long.
  • Safe key rotation.RotateEncryption re-encrypts vendor keys and node tokens in one locking transaction, and the full rotation runs against Postgres in CI.
  • Signed releases. Images andchecksums.txt are signed with keyless cosign; images carry an SBOM and SLSA provenance;govulncheck runs on every pull request.
  • Apache 2.0 instead of MIT, which adds an explicit patent grant.

The neighbours #

ktms-admin manages models, rules, keys and vendor credentials in the browser, and has a playground for trying a prompt through your rules. Sign-in is a password, Google or any OpenID Connect provider, GitHub, or the access proxy you already run, limited to an allowlist.

light-tokenmeter answers “who is spending our tokens, and are they worth it?”: one event per call in, summaries per project, user, action, vendor and model out. The console labels each group greedy or worthy from those summaries.

Try it #

git clone https://github.com/ai-process/ktms.git && cd ktms
OPENAI_API_KEY=sk-... docker compose up -d --build
curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer ktms_dev_default_key" -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Say hi in three words."}]}'

Upgrading from 0.14.x is one step. From older 0.x releases, upgrade to 0.14.x first.

What 1.0 promises #

0.x made no compatibility promise. From 1.0 on, nothing below changes incompatibly within 1.x:

Surface What stays compatible
gRPC ktms.v1 No RPC, field, enum value or message is removed or renamed; field numbers and types stay.
HTTP /v1 andX-KTMS-* headers Paths, request and response fields, header names.
Environment variables Names, defaults and meaning.
Database Any 1.x release migrates the database of an earlier one by itself. No manual SQL.
Metrics, usage events, node agent ktms_* names and labels,usage.v1 fields, agent and gateway compatible both ways.
Error contract Status codes and reason strings keep their meaning.

Some features work and are supported but may still change in a minor release, and say so with an Experimental: comment in the proto: the judge’s effort choice for model: "auto", shadow traffic and its evaluation, the semantic cache and latent guardrails, JudgeStream and EvaluateRollouts, and the YAML configuration format and the ktms command until the format reaches v1.

How the promise is checked

A promise nobody tests is a wish. Every pull request and every release runs two gates:

  • buf breaking compares the proto with the previous release.
  • An upgrade test starts the previous release and the oldest release of the same major, lets them write a database, migrates it with the new build, checks that the new build serves it, and then runs the old release again on the migrated database. That last step is what makes rolling upgrades safe.

The release workflow publishes binaries only after both pass, and Docker images only after the binaries.

Next #

Routing sensitive data to your own nodes only, hedged requests, a corrective retry for JSON, and a Grafana dashboard. None of it breaks the 1.x API.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @keystone. the model switchboard. 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/keystone-1-0-what-st…] indexed:0 read:7min 2026-10-27 · —