{"slug": "open-source-llm-firewall-with-reversible-pii-scrubbing-and-finops-routing", "title": "Open-source LLM Firewall with reversible PII scrubbing and FinOps routing", "summary": "A developer released Sentinel AI Gateway, an open-source proxy that sits between internal applications and external LLM providers such as OpenAI, Anthropic and AWS Bedrock to scrub personally identifiable information and route requests by cost. The gateway strips emails, credit card numbers, SSNs and API keys into an encrypted in-memory vault before prompts leave the network, then restores the data in responses, and it claims up to a 45% reduction in monthly LLM inference spend through semantic caching and complexity-based routing to a local Llama-3 8B model versus cloud GPT-4o or Claude. The project advertises a sub-2ms regex DLP engine, Microsoft Presidio NER integration, Redis vector semantic caching, RBAC and Prometheus/Grafana metrics, and deploys via Docker in under 30 seconds on port 8000.", "body_md": "**Enterprise-Grade LLM Firewall, Zero-Trust PII Scrubber & FinOps Routing Engine**\n\n**Sentinel AI Gateway** is a lightweight, ultra-low-latency proxy designed for enterprises adopting Generative AI. It sits between your internal microservices/employees and external LLM providers (OpenAI, Anthropic, AWS Bedrock), solving the two biggest blockers in Enterprise AI: **Data Leakage (DLP)** and **Uncontrolled API Costs (FinOps)**.\n\n``` php\ngraph LR\n    A[Internal App / Employee] -->|Raw Prompt with Secrets| B(Sentinel Gateway)\n    subgraph Zero-Trust Corporate Perimeter\n    B --> C{PII & Secret Scrubber}\n    C -->|Vaults PII Locally| D[(Encrypted In-Memory Vault)]\n    C -->|Sanitized Prompt| E{Semantic Cache & FinOps Router}\n    end\n    E -->|Cache Hit: $0 Cost| B\n    E -->|Low Complexity| F[Local Llama-3 8B]\n    E -->|High Complexity| G[Cloud GPT-4o / Claude]\n    F --> D\n    G --> D\n    D -->|Restores PII into Response| A\n```\n\n| Enterprise Challenge | Traditional Approach | Sentinel AI Gateway Approach | Business Impact | \n|---|---|---|---|\n| **Data Privacy (GDPR, HIPAA, SOC2)** | Blocking AI completely or relying on legal NDAs | **Automated Reversible Tokenization** : Strips emails, CC numbers, SSNs, and API keys*before* leaving the network. | **0% PII leakage** to 3rd-party model providers. | \n| **Runaway API Bills** | Hard rate-limiting developers | **Semantic Caching & Smart Routing** : Routes simple tasks to cheap local models and caches repetitive queries. | **Up to 45% reduction** in monthly LLM inference spend. | \n| **Vendor Lock-in** | Rewriting SDKs across 50+ microservices | **Unified OpenAI-Compatible Endpoint** : Switch providers in one config file. | **10x faster** migration between AI vendors. | \n\nDeploy the gateway in under 30 seconds:\n\n```\ngit clone https://github.com/67spp/sentinel-ai-gateway.git\ncd sentinel-ai-gateway\ndocker build -t sentinel-gateway .\ndocker run -p 8000:8000 sentinel-gateway\n```\n\nOnce running, explore the interactive Swagger UI documentation at `http://localhost:8000/docs`.\n\n**1. Outbound Request (from your developer):**\n\n```\ncurl -X POST \"http://localhost:8000/v1/chat/completions\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\n       \"prompt\": \"Send invoice to ceo@enterprise.com using card 4532-1111-2222-3333\",\n       \"task_complexity\": \"auto\"\n     }'\n```\n\n**2. What the External Cloud LLM Actually Sees:**\n\n`\"Send invoice to [EMAIL_REDACTED_1] using card [CREDIT_CARD_REDACTED_2]\"`\n\n**3. What Sentinel Returns to Your Application:**\n\n```\n{\n  \"response\": \"Processed securely by local/llama-3-8b-instruct: Confirmed action for Send invoice to ceo@enterprise.com using card 4532-1111-2222-3333\",\n  \"sanitized_prompt\": \"Send invoice to [EMAIL_REDACTED_1] using card [CREDIT_CARD_REDACTED_2]\",\n  \"pii_detected_and_masked\": 2,\n  \"routed_model\": \"local/llama-3-8b-instruct\",\n  \"cache_hit\": false,\n  \"estimated_cost_saved_usd\": 0.0045,\n  \"latency_ms\": 1.42\n}\n```\n\n- Sub-2ms Regex & Pattern DLP Engine\n- Reversible In-Memory Token Vault\n- Complexity-based FinOps Routing\n- Microsoft Presidio NLP integration for context-aware NER scrubbing\n- Redis Vector DB integration for cosine-similarity semantic caching\n- Role-Based Access Control (RBAC) & Prometheus/Grafana metrics export\n\n*Architected for high-compliance environments. For enterprise integration, custom deployment, or acquisition inquiries, reach out via [LinkedIn](https://github.com/67spp/sentinsentinel-ai-gateway/blob/main/@billlvic).*", "url": "https://wpnews.pro/news/open-source-llm-firewall-with-reversible-pii-scrubbing-and-finops-routing", "canonical_source": "https://github.com/67spp/sentinsentinel-ai-gateway", "published_at": "2026-11-04 18:30:45+00:00", "updated_at": "2026-11-04 18:50:45.797127+00:00", "lang": "en", "topics": ["ai-safety", "ai-tools", "large-language-models", "ai-infrastructure", "ai-agents"], "entities": ["Sentinel AI Gateway", "OpenAI", "Anthropic", "AWS Bedrock", "Llama-3 8B", "GPT-4o", "Claude", "Microsoft Presidio"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/open-source-llm-firewall-with-reversible-pii-scrubbing-and-finops-routing", "markdown": "https://wpnews.pro/news/open-source-llm-firewall-with-reversible-pii-scrubbing-and-finops-routing.md", "text": "https://wpnews.pro/news/open-source-llm-firewall-with-reversible-pii-scrubbing-and-finops-routing.txt", "jsonld": "https://wpnews.pro/news/open-source-llm-firewall-with-reversible-pii-scrubbing-and-finops-routing.jsonld"}}