{"slug": "llm-shield-proxy-zero-egress-pii-proxy-for-llm-soc-2-24mb-ram", "title": "LLM-Shield-Proxy - Zero-Egress PII Proxy for LLM SoC 2 (24MB RAM)", "summary": "Ninad Phalak released LLM-Shield-Proxy, an open-source zero-egress reverse proxy that redacts PII from OpenAI-compatible LLM API requests within a corporate VPC, re-hydrating SSE streams with sub-millisecond latency and a 24MB RAM footprint, to help enterprises achieve SOC 2 and HIPAA compliance. The proxy uses a compiled regex tier (<0.03ms) and a quantized ONNX NER model (~5-12ms) to avoid heavy NLP libraries, and includes a self-destructing TTL session vault for zero data liability.", "body_md": "SOC 2 and HIPAA compliance for LLM streams without breaking real-time latency.\n\n**LLM-Shield-Proxy** is an open-source, zero-egress middleware reverse proxy deployed directly within your corporate VPC. It intercepts OpenAI-compatible LLM API requests, redacts Personally Identifiable Information (PII) before it leaves your infrastructure, and deterministically re-hydrates real-time Server-Sent Events (SSE) chat responses with ultra-low stream latency.\n\nDesigned to unblock enterprise privacy compliance (**SOC 2 / HIPAA**).\n\nAuthor & Core Maintainer: **Ninad Phalak** (`ninadphalak@gmail.com`\n\n)\n\n```\npip install llm-shield-proxy \"uvicorn[standard]\"\ndocker run -d -p 8000:8000 \\\n  -e OPENAI_API_KEY=\"sk-your-openai-api-key\" \\\n  --name llm-shield-proxy \\\n  ghcr.io/ninadphalak/llm-shield-proxy:latest\nversion: \"3.8\"\n\nservices:\n  llm-shield-proxy:\n    image: ghcr.io/ninadphalak/llm-shield-proxy:latest\n    ports:\n      - \"8000:8000\"\n    environment:\n      - OPENAI_API_KEY=sk-your-openai-key-here\n      - REDIS_URL=redis://redis:6379/0\n    depends_on:\n      - redis\n\n  redis:\n    image: redis:7-alpine\n    ports:\n      - \"6379:6379\"\n```\n\nPoint your existing OpenAI SDK `base_url`\n\nto your local LLM-Shield-Proxy instance:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"your-openai-api-key\",\n    base_url=\"http://localhost:8000/v1\"  # Point to LLM-Shield-Proxy\n)\n\nresponse = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[\n        {\"role\": \"user\", \"content\": \"Contact Sarah Connor at sarah@example.com or 555-0199.\"}\n    ],\n    stream=True\n)\n\nfor chunk in response:\n    print(chunk.choices[0].delta.content or \"\", end=\"\")\n```\n\n| Existing Legacy Proxies | LLM-Shield-Proxy |\n|---|---|\nDestroys Real-Time SSE Streaming: Buffers entire responses before scanning, causing multi-second UI latency stalls. |\nUltra-Low Latency Streaming: Redacts and re-hydrates delta-by-delta as SSE packets stream. |\nHeavy Memory Footprint: Requires 1GB–2GB RAM for heavy spaCy or PyTorch NLP libraries. |\nUltra-Lightweight <24MB RAM: Runs on a microsecond compiled regex + quantized ONNX NER engine. |\nData Liability: Stores user PII in long-term databases. |\nZero Long-Term Storage: Self-destructing TTL session vault built for zero data liability. |\nComplex Cloud Egress: Routes data to 3rd-party SaaS inspection APIs. |\n100% Zero-Egress VPC: All scanning happens locally inside your secure corporate boundary. |\n\nLLM-Shield-Proxy delivers enterprise security through two core architectural breakthroughs:\n\nWhen streaming LLM responses, Server-Sent Events (SSE) send text in arbitrary token chunks. An SSE delta chunk might split a redacted placeholder tag directly across two network packets:\n\n**Chunk N:**`Hello [PER`\n\n**Chunk N+1:**`SON_1]! How can I help you today?`\n\nIf unbuffered, `[PER`\n\nleaks to the user's screen as raw un-hydrated text.\n\n**The Engineering Solution:** An asynchronous `SSERehydrationBuffer`\n\ntracks bracket boundaries (`[`\n\nand `]`\n\n). When an open bracket is detected near the tail of an incoming delta without a matching closing bracket, the buffer holds back the tail bytes until the completing chunk arrives. Once complete, the deterministic token is re-hydrated to its original value with zero UI jitter or streaming stalls.\n\nTo achieve sub-millisecond execution without blowing up infrastructure costs:\n\n**Tier 1 (Sub-millisecond Compiled Regex):** Scans structured secrets (SSNs, Credit Cards, Emails, Phone Numbers, IPv4/IPv6, API Keys) in**<0.03ms**.** Tier 2 (Quantized Local ONNX NER):**Uses a tiny, quantized ONNX Named Entity Recognition (NER) model to catch unstructured person names in**~5–12ms**.\n\nBy avoiding heavy NLP libraries like spaCy or HuggingFace transformers, LLM-Shield-Proxy runs inside a **24MB RAM process footprint** — making it fast, deterministic, and ideal for microservice sidecars. This means you can run dozens of proxy containers side-by-side on cheap micro-instances (like AWS `t4g.nano`\n\nor Docker Swarm/Kubernetes pods) for virtually zero RAM cost.\n\n**Zero-Egress Security:** 100% of PII scanning and re-hydration happens locally within your VPC. No prompt data or telemetry ever leaves your server.**Stateless Privacy (Self-Destructing Redis TTL):** Real PII is mapped to session-bound tokens (e.g.`Sarah`\n\n->`[PERSON_1]`\n\n) stored in an in-memory vault backed by strict Time-To-Live (TTL) expiration rules. When configured with Redis (`REDIS_URL`\n\n), vaults are shared across multi-replica clusters without building a permanent database of user PII.\n\nEmits structured JSON audit events (`app/audit.py`\n\n) directly to `stdout`\n\ncompatible with Datadog, Splunk, Elastic, Vanta, and Drata to prove compliance for **SOC 2 Type II** and **HIPAA** audits:\n\n```\n{\n  \"timestamp\": \"2026-08-04T01:48:00Z\",\n  \"event\": \"pii_redaction\",\n  \"session_id\": \"sess_8f179f3\",\n  \"path\": \"/v1/chat/completions\",\n  \"redactions_summary\": {\n    \"SSN\": 1,\n    \"EMAIL\": 2,\n    \"PERSON\": 1\n  },\n  \"compliance_status\": \"zero_egress_passed\"\n}\nflowchart TD\n    classDef client fill:#e0f2fe,stroke:#0284c7,stroke-width:2px,color:#0369a1,font-weight:bold;\n    classDef proxyEngine fill:#f8fafc,stroke:#475569,stroke-width:2px,color:#0f172a,font-weight:bold;\n    classDef piiSecurity fill:#fef2f2,stroke:#ef4444,stroke-width:2px,color:#991b1b,font-weight:bold;\n    classDef vault fill:#fffbebe,stroke:#f59e0b,stroke-width:2px,color:#92400e,font-weight:bold;\n    classDef upstream fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8,font-weight:bold;\n\n    UserApp[\"👤 User Application\\n(OpenAI / LangChain SDK)\"]:::client\n\n    subgraph SecurityMoat [\"🛡️ Zero-Egress Local Environment (Apache 2.0 Licensed)\"]\n        direction TD\n        FastAPIProxy[\"⚡ FastAPI Catch-All Proxy\\n(/{path:path})\"]:::proxyEngine\n\n        subgraph CascadeEngine [\"🔒 Two-Tier PII Cascade Engine\"]\n            Tier1[\"Tier 1: Compiled Regex\"]:::piiSecurity\n            Tier2[\"Tier 2: Quantized ONNX NER\"]:::piiSecurity\n            Tier1 --> Tier2\n        end\n\n        VaultStore[(\"🔑 Session Vault Store\\n(Deterministic Tokens)\")]:::vault\n        LookaheadBuffer[\"⏱️ Sliding-Window Lookahead Buffer\\n(Prevent SSE Tag Leaks)\"]:::proxyEngine\n        Rehydrator[\"🔄 Stream Re-hydrator\\n(Token -> Original Value)\"]:::proxyEngine\n    end\n\n    UpstreamLLM[\"☁️ Upstream LLM Provider\\n(OpenAI / Anthropic / vLLM)\"]:::upstream\n\n    %% Inbound Flow (Prompt Sanitization)\n    UserApp -- \"1. Inbound Raw Prompt Payload\" --> FastAPIProxy\n    FastAPIProxy -- \"2. Scan Payload\" --> Tier1\n    Tier2 -- \"3. Store Vault Keys\" --> VaultStore\n    Tier2 -- \"4. Redacted JSON Payload\" --> UpstreamLLM\n\n    %% Outbound Flow (Streaming De-redaction)\n    UpstreamLLM -. \"5. Raw SSE Stream Deltas\" .-> LookaheadBuffer\n    LookaheadBuffer -- \"6. Tag-Safe Assembly\" --> Rehydrator\n    Rehydrator <--> VaultStore\n    Rehydrator -. \"7. Sanitized Real-Time Stream\" .-> UserApp\n\n    style SecurityMoat fill:#f8fafc,stroke:#0284c7,stroke-width:2px,stroke-dasharray: 5 5,color:#0f172a\n    style CascadeEngine fill:#ffffff,stroke:#cbd5e1,stroke-width:1px\n```\n\n**Intercept:** Your application sends a standard OpenAI / LangChain payload to`localhost:8000`\n\n.**Cascade Redaction:** The proxy intercepts the JSON and routes text through a high-speed compiled Regex engine (SSNs, emails, credit cards), falling back to a local ONNX model for unstructured names.**Vault Storage:** The original PII is mapped to a deterministic tag (e.g.,`[PERSON_1]`\n\n) and stored locally in a TTL-backed session vault.**Clean Egress:** A 100% sanitized payload is forwarded to OpenAI. OpenAI never sees your raw sensitive data.\n\n**SSE Stream Intercept:** OpenAI streams the response back chunk-by-chunk via Server-Sent Events (SSE).**Lookahead Buffer:** Because tags can be split across SSE chunks (e.g.,`[PER`\n\nin chunk N and`SON_1]`\n\nin chunk N+1), the proxy's sliding-window buffer holds back unclosed brackets to prevent tag leaks.**Re-hydration:** Once a tag is fully assembled, the proxy swaps the real data back from the local vault and streams the final, un-redacted text to the user's application in real-time.\n\nLLM-Shield-Proxy is engineered for sub-millisecond overhead and ultra-lightweight resource usage. Measured over 1,000 production streaming iterations:\n\n| Metric | Average Latency | Median Latency | Footprint / Notes |\n|---|---|---|---|\nTier 1 Regex Overhead |\n`0.0294 ms` |\n`0.0291 ms` (`29.10 µs` ) |\nMicrosecond pattern scan |\nTier 2 NER Overhead |\n`0.0033 ms` |\n`0.0032 ms` (`3.20 µs` ) |\nQuantized local NER scan |\nTotal SSE Stream Overhead |\n`0.0010 ms` |\n`0.0010 ms` (`0.97 µs` ) |\nAdded latency per SSE delta chunk |\nProcess RAM Footprint |\n- | - | `24.55 MB` Resident Set Size |\n\nTo run the automated benchmark suite locally:\n\n```\npy tests/benchmark.py\n```\n\nTransparency is critical for security tooling. Please be aware of the following current limitations:\n\n**Text Only:** The proxy does not currently scan or redact text embedded inside base64 image payloads (e.g., OpenAI Vision models).**Supported Languages:** The Tier-2 ONNX NER model is currently optimized for English-language entities.**Non-Standard Streaming:** Designed for standard Server-Sent Events (SSE). Custom or proprietary streaming protocols may bypass the sliding-window buffer.\n\nRun the full automated test suite:\n\n```\npy -m pytest tests/\n```\n\nDesigned for zero-friction adoption by DevOps, Site Reliability Engineers (SREs), and Network Administrators:\n\nBuilt-in liveness and readiness endpoints return `HTTP 200 OK`\n\nfor Kubernetes, Docker Swarm, or AWS ECS health monitors:\n\n```\ncurl http://localhost:8000/health\n# Output: {\"status\":\"ok\",\"service\":\"llm-shield-proxy\",\"version\":\"1.0.4\"}\n\ncurl http://localhost:8000/livez\n# Output: {\"status\":\"ok\",\"service\":\"llm-shield-proxy\",\"version\":\"1.0.4\"}\n```\n\n100% compliant with 12-factor app standards. All upstream target routing and API keys are injected via environment variables or a `.env`\n\nfile without code modifications:\n\n`UPSTREAM_BASE_URL`\n\n: Base target URL (e.g.`https://api.openai.com`\n\nor internal`vLLM`\n\nserver).`OPENAI_API_KEY`\n\n: Upstream API key passed to target providers.`REDIS_URL`\n\n: Optional Redis connection string for distributed multi-instance session caching.\n\nLLM-Shield-Proxy runs completely stateless by default. For high-volume enterprise deployments, instances scale horizontally behind edge proxies (NGINX, Traefik, AWS ALB):\n\n```\ndocker-compose up -d --scale proxy=5\n```\n\nWhen configured with `REDIS_URL`\n\n, session vaults are shared across all proxy replicas, ensuring seamless session isolation across multi-instance clusters.\n\nEvery published release includes automated SHA-256 checksums (`checksums.txt`\n\n) and GPG detached signatures (`checksums.txt.asc`\n\n) signed by maintainer **Ninad Phalak**. You can verify checksums and cryptographic authenticity before deployment using:\n\n```\n# 1. Verify SHA-256 Checksums (Linux / macOS):\nsha256sum -c checksums.txt\n\n# On Windows (PowerShell):\nGet-FileHash llm-shield-proxy-source-v1.0.4.zip -Algorithm SHA256\n\n# 2. Verify Cryptographic GPG Signature:\ngpg --verify checksums.txt.asc checksums.txt\n```\n\nCurrently, LLM-Shield-Proxy's Tier 1 Regex engine is optimized for North American PII (US SSNs, Phone Formats). To support global GDPR compliance, I am actively looking for contributors to help expand regex payloads and Tier 2 ONNX models for:\n\n**European Formats:** UK NIN, EU Phone Numbers, IBANs.**APAC Data Structures:** India Aadhaar, APAC localized identifiers.**Multilingual NER ONNX Models:** Multilingual entity recognition models.\n\nIf you want to contribute to enterprise AI security, check out [CONTRIBUTING.md](/ninadphalak/LLM-Shield-Proxy/blob/main/CONTRIBUTING.md) and claim a locale!\n\nI am committed to maintaining LLM-Shield-Proxy as the fastest ultra-low latency redaction engine for LLMs. Here are the core architectural optimizations planned for upcoming releases — contributions and PRs are warmly welcomed:\n\n-\n**ONNX Thread Tuning (Preventing CPU Contention)*** Problem:*By default, ONNX Runtime attempts to use every available CPU core. In FastAPI, this competes with the event loop handling thousands of concurrent connections.*The Fix:*Restrict ONNX by setting`sess_options.intra_op_num_threads = 1`\n\n. This forces ONNX execution onto a single thread, keeping CPU cores free for FastAPI's event loop to stream packets instantly.\n\n-\n**Persistent Connection Pooling (The TLS Trick)*** Problem:*Opening a new TLS/SSL connection to OpenAI per request adds 50–100ms latency.*The Fix:*Maintain a persistent`httpx.AsyncClient`\n\nHTTP/2 connection pool on server startup. The proxy opens pre-warmed secure tunnels, routing requests instantly with zero TLS setup overhead.\n\n-\n**Swap to**`orjson`\n\nfor Chunk Parsing*Problem:*In an SSE stream, standard Python`json.loads`\n\nparses hundreds of delta chunks per second.*The Fix:*Swap built-in`json`\n\nfor`orjson`\n\n(written in Rust). It parses streaming LLM chunks up to 10x faster, dropping proxy overhead to near zero.\n\n-\n**Cythonize the Sliding-Window Buffer*** Problem:*The sliding-window buffer performs frequent string slicing and bracket matching.*The Fix:*Use Cython or`mypyc`\n\nto compile`streaming.py`\n\ndirectly into a C-extension binary module. Retains Python readability while executing string operations at native C speed.\n\nI am actively working with enterprise security teams to map out advanced compliance features. If your startup or organization is using LLM-Shield-Proxy to unblock LLM streaming or pass SOC 2/HIPAA audits, I would love to hear from you.\n\nEmail the core maintainer at [ninadphalak@gmail.com](mailto:ninadphalak@gmail.com) to share your feedback, request a feature, or feature your team as a case study.", "url": "https://wpnews.pro/news/llm-shield-proxy-zero-egress-pii-proxy-for-llm-soc-2-24mb-ram", "canonical_source": "https://github.com/ninadphalak/LLM-Shield-Proxy", "published_at": "2026-08-04 12:11:37+00:00", "updated_at": "2026-08-04 12:24:25.690493+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "ai-safety", "ai-policy"], "entities": ["Ninad Phalak", "LLM-Shield-Proxy", "OpenAI", "SOC 2", "HIPAA", "ONNX", "spaCy", "HuggingFace"], "alternates": {"html": "https://wpnews.pro/news/llm-shield-proxy-zero-egress-pii-proxy-for-llm-soc-2-24mb-ram", "markdown": "https://wpnews.pro/news/llm-shield-proxy-zero-egress-pii-proxy-for-llm-soc-2-24mb-ram.md", "text": "https://wpnews.pro/news/llm-shield-proxy-zero-egress-pii-proxy-for-llm-soc-2-24mb-ram.txt", "jsonld": "https://wpnews.pro/news/llm-shield-proxy-zero-egress-pii-proxy-for-llm-soc-2-24mb-ram.jsonld"}}