Top 5 LLM Gateways for Enterprises

Choosing an LLM gateway has become an infrastructure decision. Once several teams call several providers, direct integrations stop scaling: each team ends up with its own API shapes, rate limits, outage handling, and token bill. An LLM gateway sits between applications and providers to unify routing, enforce governance, and give the platform team one place to look. Bifrost, the open-source gateway Maxim AI built in Go, anchors this comparison; LiteLLM, Kong, Apache APISIX, and Azure API Management cover the other ways enterprises solve the same problem. The comparison stays on five fundamentals, and where a gateway could be self-hosted I ran it in Docker against two mock upstreams, one healthy and one returning 503 on every call.

TL;DR

An LLM gateway earns its place by doing these before it does anything clever.

One API in front of many providers. The application sends an OpenAI-shaped request and names a model. The gateway holds the provider credentials and the provider-specific request format. The test is whether an existing OpenAI SDK call works after changing only the base URL.

Failover you can see. When the primary provider fails, the gateway tries the next one, and the response says that it did. A fallback that happens silently is a fallback nobody will notice when it starts happening on every request.

Spend control per consumer. Each team or application gets its own key. The key carries a budget, a request limit, a token limit, and a list of models it may call. The gateway refuses the request that would exceed them, with a status code the client can act on.

Telemetry where you already look. Request counts, latency, and token usage as Prometheus or OpenTelemetry metrics, plus a per-request log that names the consumer. Whether prompt and completion text goes into that log should be a switch, not a surprise.

A place to run. Self-hosted or managed, and if self-hosted, what state it needs. A gateway that runs on one container with an embedded database is a different operational commitment from one that needs PostgreSQL and Redis before the first key can be issued.

Apache 2.0, written in Go, 9.2k stars on GitHub. It puts 10,000+ models across 25+ providers behind one OpenAI-compatible API, with SDK adapters for OpenAI, Anthropic, Bedrock, Google GenAI, LangChain, and LiteLLM that work through a base-URL change.

What stands out: the fundamentals are all in the free container, and they are visible.

I pulled maximhq/bifrost:latest (v2.3.2, built 20 October 2026) and started it with nothing but a volume. The startup log registered seven plugins with no configuration:

plugin status: telemetry - active
plugin status: prompts - active
plugin status: logging - active
plugin status: governance - active
plugin status: routing - active
plugin status: compat - active
plugin status: model-catalog-resolver - active

A change from early October 2026 is worth knowing before you script a deployment. Since v2.2.5 a new install needs a setup token, passed through BIFROST_SETUP_TOKEN or config.json, before the first admin account can be created, and since v2.2.6 the admin API and /metrics are locked until that account exists. Without the token, every /api call returns 403 with a message explaining why. That is the right default for something that will hold provider keys, and it is a small surprise for anyone with a curl script written before October.

With the primary provider pointed at the failing mock and a second provider at the healthy one, a request carrying "fallbacks": ["backup/gpt-4o-mini"] returned 200 from the backup, and the response said so in both the body and the headers:

X-Bifrost-Provider: backup
X-Bifrost-Routing-Info-Is-Fallback: true
X-Bifrost-Routing-Info-Primary-Provider: openai
X-Bifrost-Fallback-Index: 1

The same request without the fallback array returned the upstream's 503 unchanged. Each fallback provider gets its own retry budget, 429s rotate to another key for the same provider, and credential or quota failures rotate immediately without counting against the retry limit (see the fallback docs). Routing in the open-source build is weighted load balancing across keys, the per-request fallback chain, and key rotation; adaptive routing, which picks a provider and key from live health, is Enterprise.

Spend control is the virtual key. I created one with a request limit of three per minute and a model allowlist. The fourth call in a minute returned 429 with the reason in the body:

Rate limit exceeded: rate limit violated for vk_model_config *:
[request limit exceeded (3/3, resets every 1m)]

A call to a provider outside the allowlist returned 403 (Provider 'openai' is not allowed for virtual key 'vk-team-a'). Budgets in dollars, token limits, and team and customer hierarchies sit on the same object, and the request log recorded virtual_key_name on each entry. All of this ran against the bundled SQLite store, with no external database. SQLite is single-host, so a multi-node deployment moves the config and log stores to PostgreSQL, which open-source Bifrost supports; running those nodes as a cluster is Enterprise, as are SSO, RBAC, guardrails, and audit logs.

Telemetry is Prometheus at /metrics (bifrost_success_requests_total, bifrost_upstream_latency_seconds, bifrost_input_tokens_total, bifrost_provider_key_up, and so on) and OpenTelemetry export, with the Datadog connector, the Kafka and Pub/Sub trace connectors, and scheduled log exports to S3, GCS, and Azure Blob on Enterprise. Prompt and completion content is stored by default and can be switched off globally, per virtual key, or per request; metrics never carry content.

Best for: Enterprises running AI workloads where performance, scale, and reliability are not negotiable. Bifrost is a central gateway that routes, governs, and secures AI traffic across models and environments with little added latency, and it folds LLM gateway, MCP gateway, and agent gateway into one platform. It is built for regulated industries and strict enterprise requirements: air-gapped, VPC-isolated, and on-prem deployment, full control over data, access, and execution, and policy enforcement and governance in the gateway itself.

Python library and proxy, MIT licensed outside the enterprise/ directory, 62k stars. The README says 100+ providers; the homepage says 140+ providers and 1,892 models.

What stands out: routing depth and provider breadth, with the operational cost that comes with them.

The router picks a deployment by simple shuffle, least busy, latency, usage, or cost, with cooldowns after a configurable number of failures and fallbacks that can be scoped to context-window overflows or content-policy rejections. I configured a model group pointing at the failing mock with a fallback to the healthy one. The request succeeded, and the x-litellm-model-api-base header named the backup's URL, so the client can tell which deployment served it. The x-litellm-overhead-duration-ms header reports the proxy's own contribution per request, which measures overhead on your traffic rather than on a benchmark rig.

Two defaults changed in 2026 and both are improvements. The proxy (main-latest, version 1.106.2) now refuses to start without a master key, with a log line explaining that every request would otherwise be accepted unauthenticated. And /metrics requires a key by default; with one, the Prometheus callback exposed 60 distinct litellm_* series after a single request, covering requests, spend, tokens, deployment state, cache hits, and remaining budgets per key and team. Message content goes to logging callbacks by default and is removed with turn_off_message_logging.

The fundamentals that touch state need a database. Calling /key/generate without one returned a 500 whose message is the production architecture in one sentence: "DB not connected. This endpoint needs a database; set DATABASE_URL to a PostgreSQL connection string." The production guide adds Redis "as soon as you run more than one proxy instance" and a Redis transaction buffer around 1,000 requests per second.

Supply chain is now part of the evaluation. LiteLLM's own posts document compromised PyPI releases on 24 March 2026 (1.82.7 and 1.82.8) and 21 October 2026 (1.107.0, including the latest and main-latest Docker tags for about two and a half hours). The October root cause was a credential committed to a public repository in May and never revoked. The Docker image I pulled on 26 October was the restored 1.106.2, which the vendor recommends pinning. Enterprise features (SSO beyond five users, RBAC, audit logs, JWT auth, tag-based budgets) are an annual license sized to request capacity, with no published price. The Rust core, benchmarked at 0.7 ms p99, is beta and covers a subset of endpoints and providers; the Python path handles the rest.

Best for: Teams for whom provider coverage is the deciding constraint, who already run PostgreSQL and Redis, and who have a process for pinning and verifying the artifacts they deploy.

Kong Gateway plus AI plugins. The open-source line is 3.9.4 (October 2026); Enterprise is 3.16.0.1; and since July 2026 "Kong AI Gateway 2.x" is a separate Konnect-hosted control plane with its own entity model.

What stands out: the LLM path uses the same consumers, routes, and plugins as every other API.

If Kong already terminates your traffic, an LLM is one more upstream. The ai-proxy plugin normalizes 16 providers (OpenAI, Azure, Bedrock, Anthropic, Gemini, Mistral, Cohere, Ollama, vLLM, and others) to the OpenAI request and response format and handles chat, embeddings, responses, audio, image, and realtime route types. I ran kong:latest in DB-less mode with ai-proxy pointed at the healthy mock, and an OpenAI-shaped request came back unchanged. Token usage goes into Kong's log plugins with log_statistics, the Prometheus plugin emits ai_llm_requests_total, ai_llm_tokens_total, and ai_llm_cost_total with ai_metrics: true, and the OpenTelemetry plugin follows the Gen AI semantic conventions. None of that is license-gated.

Three of the five fundamentals sit on the other side of the license line. The 3.9.4 image reports 45 plugins; six are ai-*, and ai-proxy-advanced is not among them. Parsing a config that references it fails with "plugin 'ai-proxy-advanced' not enabled". That plugin is where load balancing (round-robin, consistent-hashing, least-connections, lowest-latency, lowest-usage, semantic, priority), failover, and the circuit breaker live. Token-based rate limiting (ai-rate-limiting-advanced), semantic caching, PII sanitizing, and the MCP proxy are also Enterprise, per the plugin hub. Kong's pricing page lists Plus from $25 per month (one million requests included, up to five LLMs) and Enterprise at custom pricing, with RBAC, SSO, and audit logging marked Enterprise only. The 2.x AI Gateway objects exist only in Konnect; self-hosted Kong gets them converted back into services, routes, and ai-proxy-advanced through deck.

Best for: Organizations already standardized on Kong Enterprise or Konnect that want LLM traffic under the same consumers, plugins, and audit trail as the rest of their APIs, and are pricing the Enterprise tier anyway.

Apache 2.0, Lua on OpenResty, 17.3k stars, an Apache Software Foundation project with API7 as the commercial vendor behind it. The current release is 3.19.0 (September 2026). Twelve ai-* plugins are in the default plugin list of the open-source build, with no license gate between them and the rest of the gateway.

What stands out: every fundamental is in the Apache-licensed build, which makes it the natural comparison with Kong.

I ran apache/apisix:latest (3.19.0) in standalone mode, where routes live in a YAML file and no etcd is needed. ai-proxy with provider: openai-compatible and override.endpoint pointed at the healthy mock returned the OpenAI-shaped response unchanged. The provider list has ten entries (openai, azure-openai, anthropic, bedrock, gemini, vertex-ai, deepseek, openrouter, aimlapi, and openai-compatible), and since 3.17 the plugin detects the inbound format (OpenAI chat, Responses, embeddings, Anthropic Messages, Bedrock Converse) and translates it to the provider's native one.

Failover is ai-proxy-multi. Two instances, the failing mock at priority 1 and the healthy one at priority 0, with fallback_strategy: ["http_429", "http_5xx"] and max_retries: 1, returned 200 from the backup on every call. The client cannot tell from the response: no header names the instance. The gateway's own logs show it twice over. The error log records ai instance primary returned status 503, falling back to backup with the upstream's error body, and the access log line lists both upstreams and both statuses:

"POST /multi HTTP/1.1" 200 ... mock-fail:32000, mock-ok:32000 503, 200

Without a fallback strategy the plugin returned the upstream's 503 with X-APISIX-Upstream-Status: 503. Active health checks per instance are an option on top, and 3.19 added fallback_http_statuses for credential errors such as 401 and 402.

Spend control is ai-rate-limiting on a route or a consumer: a token budget per time window, counted from the usage block in the response, with limit_strategy of total, prompt, or completion tokens. With key-auth on the route, a limit of 10 total tokens per minute, and a mock that reports 6 tokens per call, the third call returned 429 (the default rejection code is 503; I set it to 429), and every response carried X-AI-RateLimit-Limit, X-AI-RateLimit-Remaining, and X-AI-RateLimit-Reset headers. A request without an API key was refused with 401 before the plugin ran. Per-instance limits inside ai-proxy-multi can move a consumer to the next instance once one instance's quota is spent.

Telemetry is the standard prometheus plugin, which since 3.14 emits apisix_llm_prompt_tokens, apisix_llm_completion_tokens, apisix_llm_latency (with type="total" and type="ttft"), and apisix_llm_active_connections, labeled by route, consumer, instance ( node), and model. After the fallback test the token counters sat on node="backup", which is where the tokens were spent. logging.summaries adds model, duration, and token counts to the access log for the twenty or so log plugins; logging.payloads adds prompt and completion bodies. Both are off by default.

Rate-limit counters are per node by default; the docs say "the effective quota grows with node count", and a Redis policy for a cluster-wide quota arrived in 3.18. The admin plane is an API key with two roles (admin and viewer) and an IP allowlist. Per-operator RBAC, SSO, and a configuration audit trail are in API7 Enterprise, the commercial product, and the OpenID Connect and SAML plugins protect proxied routes rather than the admin API. The project published around two dozen plugin CVEs in 2026, concentrated in the auth plugins and all fixed by 3.19.0, so staying on the current release is part of running it. No published benchmark covers the AI plugins. Dollar budgets, per-consumer model allowlists, and a key-management UI are not in the open-source gateway; token limits and consumer restrictions are what it offers.

Best for: Teams that want an LLM gateway with Kong-style routes, consumers, and plugins, with the AI features under the same Apache license as the rest, and a platform team prepared to run the gateway, its secrets, and its admin plane.

Not a separate product. Microsoft's AI gateway is a set of policies and backend features on the API Management gateway, and the documentation says so directly: it "extends API Management's existing API gateway; it's not a separate offering." Everything here is from Microsoft Learn; I did not deploy an instance.

What stands out: spend control and failover are first-class policies, and the identity model is Entra's.

llm-token-limit enforces tokens per minute and a token quota per hour, day, week, month, or year, keyed by any policy expression, with 429 on rate and 403 on quota, and it works with OpenAI Chat Completions and Responses, Anthropic Messages (v2 tiers), and Vertex AI schemas. llm-emit-token-metric sends prompt, completion, and total tokens to Application Insights with up to five dimensions such as subscription or product. Backend pools hold up to 30 backends with round-robin, weighted, or priority-based distribution, and the circuit breaker trips on configurable status-code ranges for a configurable duration, honoring the Retry-After header that Azure OpenAI returns with its 429s. Lower-priority groups serve only when every backend above them has tripped. llm-content-safety sends prompts, completions, and MCP tool calls to Azure AI Content Safety and returns 403 on a hit.

Identity needs no gateway-specific layer. Calls to Azure OpenAI use the instance's managed identity, so no provider key exists in the application. Callers present subscription keys or Entra tokens validated by validate-azure-ad-token. Workspaces give each team its own APIs, products, and subscriptions under Azure RBAC, on Premium, Premium v2, and now Basic v2 and Standard v2.

Telemetry lands in Azure Monitor as the ApiManagementGatewayLlmLog table, with token counts and model per request and optional prompt and completion text, logged in 32 KB chunks up to 2 MB per message. MCP servers can be exposed from any REST API or proxied from an existing server, with OAuth 2.1 and content safety applied.

The "one API" fundamental is the weakest of the five. Outside the unified model API, which is in preview and translates only between OpenAI Chat Completions and Anthropic Messages, each provider is imported on its own schema; Bedrock is a passthrough API with a policy that computes the SigV4 signature in C# expressions. Token limits are counted per gateway, so a multi-region or workspace-gateway deployment does not aggregate them, and the docs describe circuit breaker and load balancing as "approximate" because gateway instances do not synchronize. Semantic caching needs an Azure Managed Redis instance created with the RediSearch module. Enabling Application Insights logging cost "a 40%-50% reduction in throughput when request rate exceeded 1,000 requests per second" in Microsoft's own load tests. Self-hosting is a container on the Developer and Premium tiers that sends no resource logs to Azure Monitor and keeps its own rate-limit counters. VNet injection is Developer, Premium, and Premium v2; list prices from the retail pricing API are roughly $150 per month for Basic v2, $700 for Standard v2, and $2,800 for Premium v2 in East US, before Redis or Content Safety. The AI Gateway tier announced in July 2026 is a preview with no SLA.

Best for: Azure-first organizations whose models are in Azure OpenAI or Foundry and whose identity is Entra, who want token limits, failover, and LLM logs without introducing a product outside the Azure control plane.

Bifrost: you want every fundamental running on one container today, PostgreSQL when a second node arrives, and clustering from Enterprise when the pilot becomes production.

LiteLLM: provider breadth decides it, you run PostgreSQL and Redis already, and you will pin releases and verify image signatures as a matter of policy.

Kong: Kong Enterprise is already your API platform. Budget for the AI plugins and treat the open-source build as a proxy, not a gateway.

Apache APISIX: you want what Kong's Enterprise AI plugins do under an Apache license, and you have a platform team to run the gateway and its admin plane.

Azure API Management: your models, identity, and monitoring are already in Azure, and multi-provider translation is a nice-to-have rather than a requirement.

Four of the five fundamentals are cheap to check in an afternoon with two mock upstreams, one of them failing. The fifth, where it runs, is the one that decides the budget: Bifrost, Kong OSS, and APISIX start as a single container, LiteLLM starts as a container plus two stateful services, and Azure starts as an account.

Disclosure: this post was made in collaboration with the Bifrost team; the testing and conclusions are mine.