# I built an open-source AI gateway for Kubernetes: budgets, kill switch and evals in CI

> Source: <https://dev.to/sandhil_kypa_b7cd7c3d5f39/i-built-an-open-source-ai-gateway-for-kubernetes-budgets-kill-switch-and-evals-in-ci-5fhf>
> Published: 2026-11-21 10:09:19+00:00

**Recommendation:** Route LLM traffic through a gateway with per-team budgets and a kill switch, and gate model changes on a shared eval suite that fails CI below an 80% quality threshold.
More in the playbook "Controlling LLM cost": <https://wpnews.pro/playbooks/llm-cost>

Every team wants AI. Then three questions arrive: What does it cost? Where are the keys? Can we stop it?

I'm a platform engineer, and I built SanVi to answer all three on Kubernetes.

Apps call one OpenAI-compatible endpoint (LiteLLM engine) with their own virtual key. Each key has its own models, budget and rate limit. No key → 401. Over the rate limit → 429. Over budget → blocked. Every call is logged with team, model, tokens and cost in Postgres.

The provider key lives only in Vault and syncs in through External Secrets. I rotated it: about 70 seconds, no redeploy.

Leaked key, runaway agent loop, spend spike: `kill_switch.sh block <team>` returns 401 in seconds, and `unblock` restores it. Nothing is deleted.

Teams, models and budgets live in `access/teams.yaml`. Changes go by pull request; CI checks policy (a greedy $50 budget was blocked); Argo CD applies on merge.

The Right-Size Advisor runs the same eval suite on each model. On my workload Haiku matched Sonnet: 95% cheaper, 40% faster. Any change that drops below 80% fails CI.

The deployment agent turns one sentence into a validated Dockerfile and Helm chart, but it can only open a pull request.

A compliance script checks 12 controls (no secrets in Git, least-privilege access, audit trail, kill switch, quality gate...) mapped to ISO 27001, SOC 2 and EU AI Act themes, on every change.

Repo, Quickstart and Helm chart: [https://github.com/sandhilk/sanvi-gateway](https://github.com/sandhilk/sanvi-gateway)

It's a personal project, so blunt feedback is welcome. What would you need before running this in your company?
