{"slug": "i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals", "title": "I built an open-source AI gateway for Kubernetes: budgets, kill switch and evals in CI", "summary": "A platform engineer built SanVi, an open-source AI gateway for Kubernetes that routes app traffic through a single OpenAI-compatible LiteLLM endpoint with per-team virtual keys, budgets, rate limits and a kill switch. The gateway logs every call's team, model, tokens and cost to Postgres, keeps the provider key in Vault synced via External Secrets, and enforces policy through pull-request-based team configs checked in CI and applied by Argo CD. A Right-Size Advisor runs the same eval suite per model — on the author's workload Haiku matched Sonnet at 95% lower cost and 40% faster — and fails CI on any change dropping below 80%.", "body_md": "Every team wants AI. Then three questions arrive: What does it cost? Where are the keys? Can we stop it?\n\nI'm a platform engineer, and I built SanVi to answer all three on Kubernetes.\n\nApps call one OpenAI-compatible endpoint (LiteLLM engine) with their own virtual key. Each key has its own models, budget and rate limit. No key → 401. Over the rate limit → 429. Over budget → blocked. Every call is logged with team, model, tokens and cost in Postgres.\n\nThe provider key lives only in Vault and syncs in through External Secrets. I rotated it: about 70 seconds, no redeploy.\n\nLeaked key, runaway agent loop, spend spike: `kill_switch.sh block <team>` returns 401 in seconds, and `unblock` restores it. Nothing is deleted.\n\nTeams, models and budgets live in `access/teams.yaml`. Changes go by pull request; CI checks policy (a greedy $50 budget was blocked); Argo CD applies on merge.\n\nThe Right-Size Advisor runs the same eval suite on each model. On my workload Haiku matched Sonnet: 95% cheaper, 40% faster. Any change that drops below 80% fails CI.\n\nThe deployment agent turns one sentence into a validated Dockerfile and Helm chart, but it can only open a pull request.\n\nA compliance script checks 12 controls (no secrets in Git, least-privilege access, audit trail, kill switch, quality gate...) mapped to ISO 27001, SOC 2 and EU AI Act themes, on every change.\n\nRepo, Quickstart and Helm chart: [https://github.com/sandhilk/sanvi-gateway](https://github.com/sandhilk/sanvi-gateway)\n\nIt's a personal project, so blunt feedback is welcome. What would you need before running this in your company?", "url": "https://wpnews.pro/news/i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals", "canonical_source": "https://dev.to/sandhil_kypa_b7cd7c3d5f39/i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals-in-ci-5fhf", "published_at": "2026-11-21 10:09:19+00:00", "updated_at": "2026-11-21 10:11:58.031416+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-agents", "ai-tools"], "entities": ["SanVi", "Kubernetes", "LiteLLM", "Vault", "External Secrets", "Argo CD", "Postgres", "GitHub"], "also_reported_by": [], "takeaway": "Routing all app LLM calls through one gateway with per-team virtual keys, budgets and a kill switch gives centralized cost control and instant revocation, and running the same eval suite per model showed Haiku matching Sonnet at 95% lower cost and 40% faster on the author's workload.", "recommendation": {"advice": "Route LLM traffic through a gateway with per-team budgets and a kill switch, and gate model changes on a shared eval suite that fails CI below an 80% quality threshold.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals", "markdown": "https://wpnews.pro/news/i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals.md", "text": "https://wpnews.pro/news/i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals.txt", "jsonld": "https://wpnews.pro/news/i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals.jsonld"}}