{"slug": "fail-closed-not-open-designing-an-ai-gateway-for-regulated-enterprises", "title": "Fail Closed, Not Open: Designing an AI Gateway for Regulated Enterprises", "summary": "A developer built Aegis Gateway, an open-source AI gateway designed for regulated enterprises that enforces governance policies before model selection. The architecture uses a three-stage pipeline—policy filtering, ranking, and fallback—to structurally prevent silent governance downgrades where non-compliant models are used during failures. The gateway refuses requests rather than routing to an unapproved model, ensuring compliance is never bypassed for availability.", "body_md": "*\"Why did this request go to that model?\"*\n\nIf your company uses LLMs across more than one team, someone will eventually ask you this question. If you work in financial services, healthcare, or anywhere else with a regulator, someone will ask it **under oath-adjacent conditions**.\n\nMost organizations can't answer it. Teams integrate directly with whatever provider they like, sensitive data handling varies by team, regional restrictions are applied inconsistently, and there's no record of why a model was selected — or, more importantly, why the alternatives weren't.\n\nI recently built [Aegis Gateway](https://github.com/abhijatchaturvedi/aegis-gateway) to explore what a proper answer looks like: a single front door for AI requests that decides which model may serve each request — and can prove why. This article is about the three design decisions that matter, and the one failure mode that I organized the entire architecture around preventing.\n\nHere's the scenario that keeps compliance officers up at night.\n\nYour gateway is well-behaved on the happy path: restricted client data goes to the on-prem model, everything's fine. Then one Tuesday the on-prem cluster goes down. Your fallback logic kicks in, does what fallback logic does — finds *something that works* — and quietly routes restricted data to an external provider. The request succeeds. Everyone's happy. Nobody notices.\n\nThat's a **silent governance downgrade**: a system that's compliant until it's inconvenient. And the insidious part is that it usually isn't a bug — it's a *reasonable-looking design* where availability and compliance compete in the same decision, and availability wins because that's what gets paged about.\n\nMy design goal was to make this failure mode **structurally impossible** — not discouraged by code review, not caught by an alert, but unreachable by construction.\n\nThe routing core has three stages with a strict, non-negotiable order:\n\n```\ncatalogue ──► Policy Engine ──► Ranking ──► Fallback Executor\n              (hard gate)       (eligible    (eligible list\n                                 models       only)\n                                 only)\n```\n\n**Stage 1 — Policy filtering.** Every model in the catalogue is checked against governance rules: Does it have the capability for this task type? Is its provider approved for this business unit? Is it certified for this data classification? Does restricted data stay on private infrastructure? Is it served in this region? Every model that fails records *the specific rule it failed*.\n\n``` php\ndef evaluate(self, model: ModelSpec, ctx: RequestContext) -> list[str]:\n    \"\"\"Return the list of violated rules (empty list = eligible).\"\"\"\n    reasons = []\n    if ctx.task_type not in model.capabilities:\n        reasons.append(f\"model does not support task '{ctx.task_type.value}'\")\n    if model.provider not in unit[\"approved_providers\"]:\n        reasons.append(f\"provider '{model.provider}' is not approved \"\n                       f\"for business unit '{ctx.business_unit}'\")\n    if req_level > CLASSIFICATION_LEVEL[model.max_classification]:\n        reasons.append(f\"model is certified only up to \"\n                       f\"{model.max_classification.value} data\")\n    # ... region residency, restricted→private, availability\n    return reasons\n```\n\n**Stage 2 — Ranking.** A weighted score over normalized cost, quality, and latency, with weights driven by the request's declared priority (Cost / Balanced / Quality). Plain arithmetic. The crucial property: **the ranker's input is the output of the policy gate.** It cannot see, and therefore cannot resurrect, a non-compliant model.\n\n**Stage 3 — Fallback.** If the selected model fails at call time, walk down the ranked list. Which list? The *eligible* list. When it's exhausted:\n\n```\nif trace.status != \"ROUTED\":\n    trace.rationale = (\n        \"No model satisfies current policy for this request context. \"\n        \"Request refused rather than downgrading governance.\"\n    )\n```\n\n`NO_COMPLIANT_ROUTE`\n\n. The gateway refuses — even if healthy external models are one function call away.\n\nThis is the whole trick, and it's almost embarrassingly simple: the compliance guarantee doesn't live in a rule that says \"don't fall back to non-compliant models.\" It lives in the **data flow**. Fallback iterates a list that non-compliant models were never in. There is no code path where the trade-off can even be expressed.\n\nAn outage is recoverable. A compliance breach is not. The architecture should not be able to make that trade, and in this design, it can't.\n\nMost systems treat decision rationale as something you grep for after the incident. Aegis treats it as part of the API contract. Every response carries a decision trace:\n\n```\n{\n  \"request_id\": \"f188db34-…\",\n  \"policy_version\": \"2026.07-r4\",\n  \"status\": \"ROUTED\",\n  \"selected_model\": \"llama-3-onprem-70b\",\n  \"fallback_model\": \"llama-3-onprem-8b\",\n  \"rejected_models\": [\n    { \"model_id\": \"chatgpt\",\n      \"reasons\": [\"Restricted data may only route to private providers\"] },\n    { \"model_id\": \"gemini-nano-banana\",\n      \"reasons\": [\"model does not support task 'text-generation'\"] }\n  ],\n  \"eligible_ranked\": [\n    { \"model_id\": \"llama-3-onprem-70b\", \"score\": 0.7,\n      \"breakdown\": { \"cost\": 0.0, \"quality\": 0.7, \"latency\": 0.0 } }\n  ]\n}\n```\n\nNotice two things.\n\nFirst, the trace distinguishes **\"not allowed\" from \"not needed.\"** When a public, cost-priority request routes to a cheap model, the premium models appear in `eligible_ranked`\n\n— they were permitted, just out-scored. When a restricted request rejects an external model, it appears in `rejected_models`\n\nwith the exact rule it broke. Those are different answers to \"why not GPT?\", and auditors care about the difference.\n\nSecond, the trace carries the **policy version**. Policies change; six months later, the audit question is \"was this decision correct *under the rules in force at the time*?\" Without versioning, that question is unanswerable.\n\nThe audit copy of the trace has one deliberate omission: the prompt. It stores a SHA-256 fingerprint and a character count — enough to prove *which* request a decision belongs to, without the audit log becoming a second, less-protected copy of your most sensitive data. Your audit trail should never be the leak.\n\nThe killer question for any multi-tenant governance system: *a second business unit wants different provider approvals, different regional restrictions, and different routing weights. How do you support that without business-unit-specific code paths?*\n\nIf your answer involves `if business_unit == \"wealth-mgmt\"`\n\n, you've already lost — every new tenant is a code change, a release, and a new place for governance bugs to hide.\n\nIn Aegis, a tenant is a policy entry:\n\n```\nbusiness_units:\n  retail-banking:\n    approved_providers: [InHouseAI, OpenAI, Anthropic, Google]\n\n  wealth-mgmt:\n    approved_providers: [InHouseAI, Anthropic]\n    weight_overrides:\n      Balanced: {cost: 0.20, quality: 0.60, latency: 0.20}\n```\n\nSame for models — a new model is one YAML entry declaring its provider, capabilities, regions, certification ceiling, and cost/quality/latency profile. The routing core never mentions a tenant or a model by name. Onboarding either one is a config change with policy review, not a deployment.\n\n(Confession: the first version of my demo UI hardcoded the business-unit dropdown. The routing core was clean, but the *presentation layer* had grown exactly the tenant-specific code path the architecture forbids. It's a good reminder that governance invariants erode from the edges — the UI, the batch job, the \"temporary\" script — not the core.)\n\nIt's 2026, so the reflexive design is \"use an LLM to pick the best model.\" I think that's exactly wrong for this layer, and it's worth being precise about why.\n\nA governance gateway's routing decision must be:\n\nRules and arithmetic give you all three. An LLM judge gives you none of them. In this architecture, LLMs are the *workload*, never the *judge*.\n\nThere's one legitimately hard sub-problem here: classifying the request itself (is this a text task or an image task? is this data really \"Internal\"?). My position: classification is an upstream contract, not a gateway guess. Applications declare task type and data classification when they integrate — it's validated, versioned, and auditable. Inferring it from prompt content would mean *reading the prompt before deciding who's allowed to read the prompt*, which is a privacy paradox you want no part of. If you must infer, do it inside the trust boundary, record the confidence in the trace, and never let the inference touch the governance rules — only the capability match. Bound the blast radius of a wrong guess to \"suboptimal route,\" never \"compliance breach.\"\n\nFull disclosure: the provider calls in the repo are simulated — `execute()`\n\nreturns canned text, and outages are injected via a test hook. That's deliberate. The simulation boundary is exactly one function; everything the project actually demonstrates — the hard gate, the ranking, the fail-closed fallback, the trace — is real, deterministic, and covered by tests, and the whole thing runs anywhere in seconds with zero API keys.\n\nThe production hardening path is the predictable one: YAML files become registry and policy services with approval workflows, the availability flag becomes health checks and circuit breakers feeding the eligibility gate, the JSONL audit file becomes an append-only store, and the business unit comes from the caller's authenticated identity instead of the request body. None of it changes the routing core — which is the point of getting the seams right.\n\nThe full implementation — routing core, Streamlit UI, FastAPI endpoint, CLI demo, and the test suite that pins the invariants — is at ** github.com/abhijatchaturvedi/aegis-gateway**. The scenario I'd point you at first: mark every compliant model as down and watch it refuse to route while healthy external models sit right there. That refusal is the entire architecture in one response.\n\n*What's your organization's answer to \"why did this request go to that model?\" I'd genuinely like to hear how others are handling this — especially the task-classification contract, which I think is the least-settled part of this space.*", "url": "https://wpnews.pro/news/fail-closed-not-open-designing-an-ai-gateway-for-regulated-enterprises", "canonical_source": "https://dev.to/abhijat_chaturvedi/fail-closed-not-open-designing-an-ai-gateway-for-regulated-enterprises-3ife", "published_at": "2026-07-26 07:46:29+00:00", "updated_at": "2026-07-26 07:59:05.217204+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-safety", "ai-policy", "developer-tools"], "entities": ["Aegis Gateway", "Abhijat Chaturvedi"], "alternates": {"html": "https://wpnews.pro/news/fail-closed-not-open-designing-an-ai-gateway-for-regulated-enterprises", "markdown": "https://wpnews.pro/news/fail-closed-not-open-designing-an-ai-gateway-for-regulated-enterprises.md", "text": "https://wpnews.pro/news/fail-closed-not-open-designing-an-ai-gateway-for-regulated-enterprises.txt", "jsonld": "https://wpnews.pro/news/fail-closed-not-open-designing-an-ai-gateway-for-regulated-enterprises.jsonld"}}