{"slug": "keystone-1-0-what-stable-means-for-a-self-hosted-llm-gateway", "title": "Keystone 1.0: what stable means for a self-hosted LLM gateway", "summary": "Keystone. The Model Switchboard. (KTMS) reached version 1.0 alongside its ktms-admin web console and light-tokenmeter usage meter, with the Go-based, Apache 2.0-licensed gateway now guaranteeing its API won't break within the 1.x line. Keystone routes service requests by effort level (LOW, MEDIUM or HIGH) plus attributes through prioritized, runtime-editable rules to model fallback chains, keeping vendor API keys encrypted in Postgres with AES-256-GCM, and idles at 7 MiB of memory. The current release, 1.3, adds a node agent for Ollama, vLLM, TGI and llama.cpp, an OpenAI-compatible HTTP API and typed gRPC ktms.v1, and an MCP server via `ktms mcp`.", "body_md": "# Keystone 1.0: what stable means for a self-hosted LLM gateway\n\nKeystone. The Model Switchboard. (KTMS) reaches 1.0 together with ktms-admin and light-tokenmeter. What the gateway does, what is now stable across 1.x, how we test that promise, and what changed on the way.\n\n**A model name in your code goes stale the day the next model ships.** With ten services, you\nrewrite it ten times, and spread vendor keys across ten configs.\n\n**Keystone. The Model Switchboard. (KTMS)** takes that out of your code. A service says how hard\nthe task is; the gateway decides which model to call, with which key, and whether your own GPU\nshould take it. It idles at 7 MiB of memory, and from 1.0 on its API doesn’t break within 1.x.\n\nThe web console **ktms-admin** and the usage meter **light-tokenmeter** reach 1.0 with it. All three\nare written in Go and licensed under Apache 2.0. The features below are as of the current 1.3.\n\n## What Keystone does\n\nKeystone sits between your services and your model vendors. Your services call one endpoint, and KTMS:\n\n- keeps the vendor API keys, encrypted in Postgres with AES-256-GCM, so clients never see them;\n- picks the model from rules, not from a name hardcoded in your code;\n- falls back to the next model when a vendor fails or throttles;\n- enforces requests per minute, daily token budgets and hard spend caps per key, user and team;\n- sends one usage event per call to a sink such as light-tokenmeter.\n\n### Effort instead of model names\n\nA service doesn’t ask for `gpt-4o`. It asks for an effort, `LOW`, `MEDIUM` or `HIGH`, and adds\nfree-form attributes such as `language` or `task`. An ordered list of rules maps that to a chain of\nmodels:\n\n```\n{\"rules\":[\n  {\"name\":\"spanish-high\", \"priority\":30, \"effort\":\"EFFORT_HIGH\",\n   \"attributes\":{\"language\":\"es\"}, \"use\":[\"gemini-2.5-pro\",\"gpt-4o\"], \"enabled\":true},\n  {\"name\":\"low-default\", \"priority\":20, \"effort\":\"EFFORT_LOW\",\n   \"use\":[\"deepseek-chat\",\"gpt-4o-mini\"], \"enabled\":true},\n  {\"name\":\"anything-else\", \"priority\":1, \"use\":[\"gpt-4o-mini\"], \"enabled\":true}\n]}\n```\n\nRules change at runtime over the admin API or the console. When a new model comes out, you edit a rule, not forty services.\n\n### Two ways in\n\nTyped gRPC (`ktms.v1`) for services, and an OpenAI-compatible HTTP API for everything that already\nspeaks OpenAI:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8080/v1\", api_key=\"ktms_<your key>\")\nreply = client.chat.completions.create(\n    model=\"auto\",\n    messages=[{\"role\": \"user\", \"content\": \"Say hi in three words.\"}],\n)\nprint(reply.choices[0].message.content, reply.model)\n```\n\nResponse headers (`X-KTMS-Vendor`, `X-KTMS-Rule`, `X-KTMS-Placement`) say who answered and why.\n\n### What it does (as of 1.3)\n\n- **Routing:** effort LOW/MEDIUM/HIGH plus attributes, prioritized rules changed at runtime, fallback chains,`model: \"auto\"` with a judge (250 ms, falls back to MEDIUM), canaries sticky per user, shadow traffic with evaluation, a dry run with`ktms route explain` .\n- **Reliability:** circuit breaker, a model paused on`Retry-After` , one attempt is one vendor call, defined behaviour when Postgres or Redis fail.\n- **Money and limits:** RPM/TPM and a model allowlist per key, daily and monthly spend caps per key and user, a monthly budget per team, prompt-cache pricing, vendor Batch APIs.\n- **Your own GPUs:** a node agent for Ollama, vLLM, TGI and llama.cpp, admission by real queue depth, “own nodes only” placement, capacity levels and load schedules, a local usage log with`ktms-node stats` .\n- **Security:** vendor keys in Postgres under AES-256-GCM with rotation, PII masking, prompt-injection detection, latent guardrails and a semantic cache (experimental).\n- **API:** typed gRPC`ktms.v1` and an OpenAI-compatible HTTP API, streaming, tool calling, lossless JSON Schema, embeddings, speech, images, Judge.\n- **Management:** the`ktms` CLI with contexts and`ktms init` , configuration as YAML (`plan` /`apply` ), an MCP server (`ktms mcp` ) for AI assistants, a Claude Code plugin, an`admin:read` scope, and the ktms-admin console.\n- **Operations:**`ktms_*` metrics with suggested alerts, a Helm chart on OCI, signed images with SBOM and SLSA provenance, an upgrade test on every pull request.\n\n## Footprint\n\nKeystone is one Go process with no interpreter. On our stand of eleven self-hosted gateways\n(Google Cloud `c3-standard-4`, 4 vCPU, mock upstream, two runs), it used about 7 MiB idle and\n20 MiB at 200 RPS, added about 0.64 ms at p50, sustained 4,000 RPS, and answered its first request\nabout 170 ms after `docker start`. agentgateway, APISIX and Kong add less latency; only agentgateway\nmatches the memory. The full write-up is in [Go vs Python for an LLM gateway](https://blog.ai-process.dev/2026/10/26/go-vs-python-llm-gateway/).\n\n## What changed on the way to 1.0\n\n- **Organizations and teams.** Team keys share a model allowlist and a monthly budget that the\ngateway enforces across all of them.\n- **Vendor calls you can reason about.** Vendor SDK retries are off, so one router attempt is one\nvendor call.`Retry-After` pauses a model for that vendor credential, and the chain moves on.\n- **Defined behaviour when a dependency fails.** With Postgres down, keys verified in the last five\nminutes keep working and routing serves from the last config snapshot. With Redis down, rate\nlimits fail open, and you choose what spend caps do. Usage events queue and are delivered once.\n- **Lossless JSON Schema.**`enum` ,`format` ,`nullable` and`additionalProperties` reach the vendor.\nWhere a vendor can’t enforce a schema, Keystone checks the reply, retries once and falls back.\n- **Local nodes by real queue depth.** The node agent reports the engine’s own queue (vLLM, TGI,\nllama.cpp), and admission skips a node where the request would wait too long.\n- **Safe key rotation.**`RotateEncryption` re-encrypts vendor keys and node tokens in one locking\ntransaction, and the full rotation runs against Postgres in CI.\n- **Signed releases.** Images and`checksums.txt` are signed with keyless cosign; images carry an\nSBOM and SLSA provenance;`govulncheck` runs on every pull request.\n- **Apache 2.0** instead of MIT, which adds an explicit patent grant.\n\n## The neighbours\n\n**ktms-admin** manages models, rules, keys and vendor credentials in the browser, and has a\nplayground for trying a prompt through your rules. Sign-in is a password, Google or any OpenID\nConnect provider, GitHub, or the access proxy you already run, limited to an allowlist.\n\n**light-tokenmeter** answers “who is spending our tokens, and are they worth it?”: one event per\ncall in, summaries per project, user, action, vendor and model out. The console labels each group\ngreedy or worthy from those summaries.\n\n## Try it\n\n```\ngit clone https://github.com/ai-process/ktms.git && cd ktms\nOPENAI_API_KEY=sk-... docker compose up -d --build\ncurl -s http://localhost:8080/v1/chat/completions \\\n  -H \"Authorization: Bearer ktms_dev_default_key\" -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"auto\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hi in three words.\"}]}'\n```\n\nUpgrading from 0.14.x is one step. From older 0.x releases, upgrade to 0.14.x first.\n\n## What 1.0 promises\n\n0.x made no compatibility promise. From 1.0 on, nothing below changes incompatibly within 1.x:\n\n| Surface | What stays compatible | \n|---|---|\n| gRPC `ktms.v1` | No RPC, field, enum value or message is removed or renamed; field numbers and types stay. | \n| HTTP `/v1` and`X-KTMS-*` headers | Paths, request and response fields, header names. | \n| Environment variables | Names, defaults and meaning. | \n| Database | Any 1.x release migrates the database of an earlier one by itself. No manual SQL. | \n| Metrics, usage events, node agent | `ktms_*` names and labels,`usage.v1` fields, agent and gateway compatible both ways. | \n| Error contract | Status codes and reason strings keep their meaning. | \n\nSome features work and are supported but may still change in a minor release, and say so with an\n`Experimental:` comment in the proto: the judge’s effort choice for `model: \"auto\"`, shadow traffic\nand its evaluation, the semantic cache and latent guardrails, `JudgeStream` and\n`EvaluateRollouts`, and the YAML configuration format and the `ktms` command until the format reaches `v1`.\n\n### How the promise is checked\n\nA promise nobody tests is a wish. Every pull request and every release runs two gates:\n\n- **`buf breaking`** compares the proto with the previous release.\n- **An upgrade test** starts the previous release and the oldest release of the same major, lets\nthem write a database, migrates it with the new build, checks that the new build serves it, and\nthen runs the old release again on the migrated database. That last step is what makes rolling\nupgrades safe.\n\nThe release workflow publishes binaries only after both pass, and Docker images only after the binaries.\n\n## Next\n\nRouting sensitive data to your own nodes only, hedged requests, a corrective retry for JSON, and a Grafana dashboard. None of it breaks the 1.x API.\n\n- Code: [github.com/ai-process/ktms](https://github.com/ai-process/ktms)\n- Docs: [ai-process.dev](https://ai-process.dev)\n- Console: [github.com/ai-process/ktms-admin](https://github.com/ai-process/ktms-admin)\n- Usage meter: [github.com/ai-process/light-tokenmeter](https://github.com/ai-process/light-tokenmeter)", "url": "https://wpnews.pro/news/keystone-1-0-what-stable-means-for-a-self-hosted-llm-gateway", "canonical_source": "https://blog.ai-process.dev/2026/10/27/keystone-1-0/", "published_at": "2026-10-27 07:00:00+00:00", "updated_at": "2026-11-04 18:51:05.550327+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "agent-protocols"], "entities": ["Keystone. The Model Switchboard.", "KTMS", "ktms-admin", "light-tokenmeter", "Ollama", "vLLM", "TGI", "llama.cpp"], "also_reported_by": [], "takeaway": "Keystone 1.0 lets services request an effort level (LOW/MEDIUM/HIGH) plus attributes instead of a hardcoded model name, with prioritized rules editable at runtime mapping to fallback chains, so swapping in a new model is a rule edit rather than a change across every service.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/keystone-1-0-what-stable-means-for-a-self-hosted-llm-gateway", "markdown": "https://wpnews.pro/news/keystone-1-0-what-stable-means-for-a-self-hosted-llm-gateway.md", "text": "https://wpnews.pro/news/keystone-1-0-what-stable-means-for-a-self-hosted-llm-gateway.txt", "jsonld": "https://wpnews.pro/news/keystone-1-0-what-stable-means-for-a-self-hosted-llm-gateway.jsonld"}}