{"slug": "show-hn-self-hosted-ai-gateway-mcp-budget-pii-smart-router-fallback", "title": "Show HN: Self-hosted AI gateway – MCP, budget, PII, smart router, fallback", "summary": "ILTER, a self-hosted AI gateway, launches as a single static executable that sits between applications and AI providers to address cost control, data privacy, and API reliability. The gateway features an MCP marketplace, smart routing with real-time complexity scoring, budget limits, PII masking, semantic caching, prompt guardrails, and agent loop detection, requiring no code changes or external dependencies.", "body_md": "Deploying AI models to production brings critical operational challenges such as unpredictable costs, data privacy risks, and API outages. **ILTER** is an independent gateway that sits between your application and AI providers; transforming your entire AI traffic into a secure, optimized, and fully controlled infrastructure without requiring any architectural changes in your code.\n\n- 🧩\n**MCP Gateway & Marketplace:** Connect AI to your APIs, CRM, or MCP servers — zero changes required. - ⚡\n**Smart Router:** Every request is scored for complexity in real time and routed to the right model tier. - 🌐\n**Smart Fallback:** Provider down or rate-limited? ILTER fails over to another key or provider, no changes. - 💰\n**Budget Control:** Hard daily/monthly spending limits per key — traffic cuts off the instant a limit is hit. - 🪪\n**PII Guard:** Emails, SSNs, and other sensitive data are masked before they ever leave your network. - 🧠\n**Semantic Cache:** Queries are served from cache — vector search with SHA-256 exact-match fallback. - 🛡️\n**Prompt Guardrails:** Blocks injection, toxic content, and off-topic requests before they reach the model. - 🌀\n**Agent Loop Detector:** Catches loops by rate, fingerprint, cost, or depth before burning budget. - ⏱️\n**Cron Engine:** Schedule recurring AI workflows with standard cron expressions — no external queue needed. - 📡\n**Observability:** Prometheus metrics and OpenTelemetry tracing, ready for Grafana, Datadog, or Honeycomb. - 📊\n**Dashboard:** Embedded web UI for KPIs, audit logs, API keys, and built-in chat playground — no server.\n\nAll this infrastructure requires no external server, Node.js/Python environment, or complex dependencies.\n\nIt comes as a zero-setup, single static executable (binary) ready to run in seconds.\n\n```\n./ilter serve\n# Proxy: http://localhost:8181/v1/chat/completions\n# Dashboard: http://localhost:9191\n# Metrics: http://localhost:9192/metrics\n```\n\nEvery business integrating AI faces uncontrolled costs, privacy concerns, and unpredictable agent behaviors. ILTER solves these directly at the gateway level.\n\nConnecting AI to your internal APIs or CRM usually requires heavy client-side modifications.\n\n**Zero Client Changes:** Automatically injects tools from registered MCP servers into every chat completion request.**Interception:** ILTER catches the`tool_call`\n\n, executes the MCP server tool locally (`stdio`\n\n/SSE), and returns the result to the model.**OpenAPI Bridge:** Convert any REST API to an MCP tool list instantly.**OAuth PKCE Support:** Standard OAuth PKCE authorization endpoints (`/.well-known/*`\n\n,`/authorize`\n\n,`/token`\n\n,`/register`\n\n) on port`8181`\n\nfor remote MCP clients.\n\nUsing expensive models for every simple question wastes engineering time and money. ILTER routes requests dynamically based on context.\n\n**Real-Time Scoring:**`<0.03ms`\n\nheuristic complexity score (0–100) based on word count, reasoning-required phrases, code blocks, and tool calls.**Automatic Tier Selection:** Routes dynamically between economy, standard, and premium models.**Custom Overrides:** E.g.,`complexity > 50 → gpt-4o`\n\nor`prompt contains \"analyze\" → claude-sonnet`\n\n.**Load Balancing:** Weighted round-robin, cost-optimized, latency-optimized, and priority-based strategies across healthy provider circuits.\n\nDon't get locked into a single vendor or disrupted by provider outages.\n\n**Single Endpoint:** Hides 9 providers behind one uniform OpenAI-compatible API.**Circuit Breaker:**`RetryTransport → CircuitBreakerTransport`\n\n. 5 consecutive failures open the circuit and trigger automatic fallback.**Seamless Failover:** Switching from OpenAI to Anthropic during an outage requires zero code changes.\n\n**Vector Search:** Redis Stack-backed similarity search to catch and serve repetitive queries.**Exact-Match Fallback:** SHA-256 caching works even without an embedding model.**Async Operations:** Cache writing does not block the primary request path.\n\n**Security:** Prompt injection detection and toxicity filtering.**Topic Control:** Keyword-based and custom regex rules to block specific topics.**Actions:** Set per-rule severity to`block`\n\n,`warn`\n\n, or`mask`\n\n.\n\nPer-token pricing creates end-of-month surprises. ILTER tracks costs in real-time for each API key and enforces hard limits.\n\n**Hard Kill Switch:** Configurable daily and monthly spending limits (USD). Returns`429 budget_exceeded`\n\nthe millisecond the limit is breached.**Warning Thresholds:** Default 80% usage alerts.**Real-time Tracking:** Cost breakdowns by model, provider, and individual keys via the dashboard.\n\nSending customer data to cloud providers violates GDPR and HIPAA. ILTER masks this data before it ever leaves your network.\n\n**Triple-Layer Engine:** Bloom Filter + Aho-Corasick Trie + Regex working together.**Detected Types:** Names (EN/TR), Emails, Phones, SSN/National IDs (TCKN), Credit Cards (Luhn validated), IPs.**Three Modes:** Mask (`[PII_EMAIL]`\n\n), Reversible (`PII:EMAIL:a8b9f1`\n\n— restored in response/SSE stream), or Block (`422 pii_blocked`\n\n).**Ultra-Fast:**`<0.04ms`\n\nlatency overhead with`<5MB`\n\nmemory footprint.\n\nAgentic workflows can enter infinite loops, generating massive bills overnight. ILTER catches them while you sleep.\n\n**Rate Limit:** Blocks if >30 requests/sec.**Fingerprint Match:** Blocks if the exact same prompt is sent >5 times in a 20-request window.**Cost Accumulation:** Blocks if >$5 is spent within 5 minutes on a single session.**Session Depth:** Throttles if a single session exceeds 100 requests.\n\nRunning periodic AI tasks (summaries, reports) usually requires external queues and workers. ILTER embeds this directly in the binary.\n\n**No External Queue:** Uses standard cron expressions (e.g.,`0 9 * * 1`\n\n).**Workflow State:** Pass data between steps using`{{.Input}}`\n\nand`{{.prev}}`\n\n.**Robust Execution:** Webhook triggers, exponential backoff, and dead-letter queues included.\n\n**Metrics:** Native Prometheus metrics on`/metrics`\n\n(Port`9192`\n\n).**Tracing:** OpenTelemetry (OTLP push) ready for Grafana Cloud, Datadog, or Honeycomb.**Audit Logs:** Full request logs, cost trends, and PII event indicators.\n\nNo npm, no build step, no separate server. A modern web UI embedded directly into the Go binary via `embed`\n\n.\n\n**Overview:** KPIs, monthly spend, routing decisions, and feature toggle hub.**Chat Playground:** Use any registered model directly from the dashboard.**Management:** Create API keys, configure thresholds, register MCP servers, and manage Cron jobs. (Accessible at Port`9191`\n\n).\n\n```\n# Requires nothing — it just works\n./ilter serve\n```\n\nFor the interactive setup wizard:\n\n```\n./ilter init\n```\n\nTo see the dashboard filled with mock costs, requests, and PII events:\n\n```\n./ilter init --demo && ./ilter serve\n```\n\nYour first real request (Point your existing OpenAI SDK to ILTER):\n\n```\ncurl -X POST http://localhost:8181/v1/chat/completions \\\n  -H \"Authorization: Bearer <ilter-api-key>\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"gpt-4o-mini\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]}'\n```\n\n| Provider | Notes |\n|---|---|\nOpenAI |\nNative format |\nAnthropic |\nSystem message extraction, content block conversion |\nGoogle Gemini |\nOpenAI-compatible mode |\nDeepSeek |\nOpenAI-compatible |\nOpenRouter |\n`HTTP-Referer` + `X-Title` headers automatically injected |\nOllama |\nLocal inference, OpenAI-compatible mode |\nQwen (Alibaba) |\nOpenAI-compatible, model name mapping |\nOpenCode |\n`opencode_go` and `opencode_zen` SDK endpoint mapping |\nMock |\nBuilt-in mock provider for local testing |\n\n```\nYour Application\n    │  OpenAI API format\n    ▼\nILTER :8181\n    │\n    ├─ Auth              — ilter-xxxx key (Argon2id/SHA-256 + LRU cache)\n    ├─ Rate Limiter      — Redis / in-memory token bucket, RPM/TPM per key\n    ├─ Budget Enforcer   — Monthly spend limit, hard reject\n    ├─ Prompt Injection  — Inject system prompts from DB\n    ├─ PII Masker        — Bloom + Aho-Corasick + Regex, <0.04ms\n    ├─ Guardrails        — Injection detection, topic blocking\n    ├─ MCP Inject        — Tool injection + tool_call interception + OAuth PKCE\n    ├─ Smart Router      — Real-time complexity scoring & tier selection\n    ├─ Loop Detector     — Rate / fingerprint / cost / session\n    ├─ Semantic Cache    — Redis Stack vector search + SHA256 fallback\n    └─ Provider Router   — Weighted round-robin, circuit breaker, fallback\n           │\n           ├─ OpenAI / Anthropic / Gemini / DeepSeek\n           ├─ OpenRouter / Ollama / Qwen / OpenCode\n           └─ ...\n\nILTER :9191  →  Dashboard (Astro + React, Go embed)\nILTER :9192  →  /metrics  (Prometheus, OpenTelemetry bridge)\n```\n\n**Language:** Go 1.26.3, single binary, CGo-free (`CGO_ENABLED=0`\n\n), goroutine concurrency.**Router:**`chi`\n\nv5.**Database:** SQLite —`modernc.org/sqlite`\n\n, pure Go, WAL mode.**Cache:** Redis Stack 7+ — optional, graceful degradation.**Config:** Compiled defaults +`ILTER_*`\n\nenv vars — no configuration file required.\n\n→ Architecture details: `docs/architecture.md`\n\n→ Design decisions & FAQ: `docs/faq.md`\n\n→ Gateway comparison: `docs/comparison.md`\n\n```\n# Single container (scratch-based image, <20MB)\n# Admin key + a provider key are enough to boot straight into `serve` — no\n# `ilter init` step needed. The SQLite DB is created on first start, and the\n# provider whose ILTER_PROVIDER_<NAME>_API_KEY is set is enabled automatically.\ndocker run -d \\\n  -p 8181:8181 -p 9191:9191 -p 9192:9192 \\\n  -v $(pwd)/data:/app/data \\\n  -e ILTER_ADMIN_API_KEY=<your-own-random-secret> \\\n  -e ILTER_PROVIDER_OPENAI_API_KEY=sk-... \\\n  ghcr.io/ilter-ai/ilter:latest\n\n# Full local stack: ILTER + Redis Stack + Ollama\ndocker compose up -d\n```\n\nWithout `ILTER_ADMIN_API_KEY`\n\nand a provider key both set, `serve`\n\nrefuses to start — there'd be no way to authenticate or route requests — and exits with a message telling you to run `ilter init`\n\nor set both.\n\nImage size: **<20MB** (3-stage build: Bun web → Go UPX → empty `scratch`\n\nbase image). This applies to the Docker image specifically — the plain binary from [Releases](https://github.com/ilter-ai/ilter/releases) (no UPX) is ~35-40MB.\n\nZero configuration by default. Three levels of overrides:\n\n| Level | Method | Example |\n|---|---|---|\nBoot |\n`ILTER_*` env vars |\n`ILTER_SERVER_PORT=9090` , `ILTER_METRICS_LISTEN_ADDR=:9192` |\nRuntime |\nDashboard or `ilter init` wizard |\nToggle PII, add providers, update routing strategies |\nPer-key |\nDashboard or Admin API | Team-based budget, rate limit, allowed models |\n\n→ All options: `docs/configuration.md`\n\n| Command | Description |\n|---|---|\n`ilter serve` |\nStart proxy + dashboard |\n`ilter init` |\nInteractive setup wizard |\n`ilter init --demo` |\nDashboard demo with mock data |\n`ilter models` |\nSupported models — provider, tier, cost |\n`ilter models --update` |\nModel discovery from all providers |\n\n```\nmake build      # Go binary + web assets\nmake check      # Build + lint (Go + web)\nmake test       # go test -race -count=1 ./...\nmake fix        # gofumpt + biome format\n```\n\nContribution guidelines: `CONTRIBUTING.md`\n\nApple M4, Go 1.26.3:\n\n| Operation | Latency |\n|---|---|\n| Core proxy overhead | <0.8ms |\n| PII Guard scan | 0.031ms |\n| Smart Router scoring | 0.022ms |\n| Loop fingerprinting | 0.008ms |\n\nApache 2.0 with Commons Clause — see [LICENSE](/ilter-ai/ilter/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-self-hosted-ai-gateway-mcp-budget-pii-smart-router-fallback", "canonical_source": "https://github.com/ilter-ai/ilter", "published_at": "2026-07-29 06:19:29+00:00", "updated_at": "2026-07-29 06:52:43.953676+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "ai-safety", "ai-products", "developer-tools"], "entities": ["ILTER", "OpenAI", "Anthropic", "Redis Stack", "Prometheus", "OpenTelemetry", "Grafana", "Datadog"], "alternates": {"html": "https://wpnews.pro/news/show-hn-self-hosted-ai-gateway-mcp-budget-pii-smart-router-fallback", "markdown": "https://wpnews.pro/news/show-hn-self-hosted-ai-gateway-mcp-budget-pii-smart-router-fallback.md", "text": "https://wpnews.pro/news/show-hn-self-hosted-ai-gateway-mcp-budget-pii-smart-router-fallback.txt", "jsonld": "https://wpnews.pro/news/show-hn-self-hosted-ai-gateway-mcp-budget-pii-smart-router-fallback.jsonld"}}