cd /news/ai-infrastructure/i-built-an-open-source-ai-gateway-fo… · home › topics › ai-infrastructure › article
[ARTICLE · art-173726] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

I built an open-source AI gateway for Kubernetes: budgets, kill switch and evals in CI

A platform engineer built SanVi, an open-source AI gateway for Kubernetes that routes app traffic through a single OpenAI-compatible LiteLLM endpoint with per-team virtual keys, budgets, rate limits and a kill switch. The gateway logs every call's team, model, tokens and cost to Postgres, keeps the provider key in Vault synced via External Secrets, and enforces policy through pull-request-based team configs checked in CI and applied by Argo CD. A Right-Size Advisor runs the same eval suite per model — on the author's workload Haiku matched Sonnet at 95% lower cost and 40% faster — and fails CI on any change dropping below 80%.

recommendation

Route LLM traffic through a gateway with per-team budgets and a kill switch, and gate model changes on a shared eval suite that fails CI below an 80% quality threshold.

more in playbook: Controlling LLM cost
by read1 min views1 publishedNov 21, 2026

Every team wants AI. Then three questions arrive: What does it cost? Where are the keys? Can we stop it?

I'm a platform engineer, and I built SanVi to answer all three on Kubernetes.

Apps call one OpenAI-compatible endpoint (LiteLLM engine) with their own virtual key. Each key has its own models, budget and rate limit. No key → 401. Over the rate limit → 429. Over budget → blocked. Every call is logged with team, model, tokens and cost in Postgres.

The provider key lives only in Vault and syncs in through External Secrets. I rotated it: about 70 seconds, no redeploy.

Leaked key, runaway agent loop, spend spike: kill_switch.sh block <team> returns 401 in seconds, and unblock restores it. Nothing is deleted.

Teams, models and budgets live in access/teams.yaml. Changes go by pull request; CI checks policy (a greedy $50 budget was blocked); Argo CD applies on merge.

The Right-Size Advisor runs the same eval suite on each model. On my workload Haiku matched Sonnet: 95% cheaper, 40% faster. Any change that drops below 80% fails CI.

The deployment agent turns one sentence into a validated Dockerfile and Helm chart, but it can only open a pull request.

A compliance script checks 12 controls (no secrets in Git, least-privilege access, audit trail, kill switch, quality gate...) mapped to ISO 27001, SOC 2 and EU AI Act themes, on every change.

Repo, Quickstart and Helm chart: https://github.com/sandhilk/sanvi-gateway It's a personal project, so blunt feedback is welcome. What would you need before running this in your company?

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @sanvi 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-built-an-open-sour…] indexed:0 read:1min 2026-11-21 · —