{"slug": "migrating-legacy-llm-infrastructure-to-an-ai-gateway", "title": "Migrating Legacy LLM Infrastructure to an AI Gateway", "summary": "A developer migrated a legacy LLM infrastructure to the open-source Bifrost AI gateway, demonstrating improved availability and cost control. The migration replaced direct provider calls with a gateway that supports fallbacks, caching, and per-team key management, reducing a 43% failure rate to zero during a simulated provider outage.", "body_md": "Your support copilot started as a weekend prototype: one model, one provider, one API key in an env var. Then it became production, and you inherited its weaknesses: the provider's availability is your availability, every retry is your code, spend is a mystery until the invoice, and agents bolt tool-use on however they can. This post migrates that stack onto an enterprise AI gateway — and actually runs the migration, with the raw outputs to show for it.\n\nThe gateway here is [Bifrost](https://www.getmaxim.ai/bifrost), an open-source ([github.com/maximhq/bifrost](https://github.com/maximhq/bifrost), Apache-2.0) gateway written in Go, presenting a single OpenAI-compatible API across 23+ providers. I rebuilt the legacy stack locally — mock providers with deterministic latency, a realistic traffic pattern — and moved it behind Bifrost step by step.\n\nA support copilot's traffic has a shape: mostly repeated FAQ-style questions, plus one-off queries. My traffic mix: 60 requests — 40 FAQ prompts (8 distinct questions asked 5 times each) plus 20 one-offs. Mock provider latency: 200 ms.\n\nRun 1, direct to the provider:\n\n```\nlegacy: 60 ok / 0 fail, 9,335 tokens billed, ~201 ms avg latency\n```\n\nRun 2 — the provider dies mid-sweep, as providers do:\n\n```\nlegacy + failure: 34 ok / 26 fail\n```\n\n26 requests — 43% — failed outright. Nothing in the legacy stack retries across providers because nothing can: the app speaks one provider's API. And availability is only the loudest problem. The quieter ones: every team's service embeds the same shared key (one key's quota is everyone's ceiling, and revoking it breaks everyone at once), there is no per-team attribution of spend, and the only way to cut cost on repeated questions is to build caching yourself — request normalization, hash keys, TTLs, invalidation — inside the application. That is the whole argument for a gateway in one row of output.\n\nSeven moves, each reversible. Diagrams follow the flow.\n\n```\ndocker run -p 8080:8080 maximhq/bifrost\n```\n\nOne config file wires your existing provider and key; the app keeps working untouched. Mine, reduced to the bones:\n\n```\n{\n  \"providers\": {\n    \"openai\": {\n      \"keys\": [{ \"name\": \"primary\", \"value\": \"mock-key\", \"weight\": 1.0,\n                 \"models\": [\"support-chat\"] }],\n      \"network_config\": { \"base_url\": \"http://provider:9001\",\n                          \"default_request_timeout_in_seconds\": 30 }\n    }\n  }\n}\n```\n\nThe [gateway setup guide](https://docs.getbifrost.ai/quickstart/gateway/setting-up) covers the web-UI alternative, and there is a [Go SDK](https://docs.getbifrost.ai/quickstart/go-sdk/setting-up) if you want the gateway embedded rather than adjacent.\n\nThe [OpenAI-compatible API](https://docs.getbifrost.ai/providers/supported-providers/overview) means the client change is a base URL — `api.openai.com`\n\n→ `localhost:8080`\n\n— not a rewrite. Every request now flows through a hop you control. Screenshot of the providers page after this step:\n\nA second provider config (`anthropic`\n\nin my bench) plus a request-level fallback chain:\n\n```\n{\n  \"model\": \"openai/support-chat\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}],\n  \"fallbacks\": [\"anthropic/support-chat\"]\n}\n```\n\nThen the proof. Healthy primary:\n\n```\n\"routing_info\": {\"provider\": \"openai\", \"key\": \"primary\", \"is_fallback\": false}\n```\n\nI killed the primary provider's process and re-sent the identical request:\n\n```\n\"routing_info\": {\n  \"provider\": \"anthropic\", \"key\": \"backup\",\n  \"is_fallback\": true,\n  \"primary_provider\": \"openai\", \"primary_model\": \"support-chat\"\n}\n```\n\nThe request succeeded on the backup and the response says exactly what happened — `is_fallback: true`\n\nwith the failed primary recorded. That audit trail is what you want at 2 a.m.: not just \"it kept working,\" but \"it kept working this way.\" The [retries and fallbacks docs](https://docs.getbifrost.ai/features/retries-and-fallbacks) cover chained fallbacks and per-provider retry counts.\n\nOne honest caveat from my bench: failover on *initial* connection-refused (provider already dead before the first connect) was inconsistent in my mock setup — it fired reliably when the upstream errored or the connection dropped mid-pool, but a cold connection-refused sometimes returned a 502 instead of failing through. Validate failover against your providers' real failure modes before you trust it in production.\n\n[Semantic caching](https://docs.getbifrost.ai/features/semantic-caching) has two modes: exact-match (direct hash, no embeddings needed) and embedding-based similarity. Config for direct mode with a Redis Stack vector store:\n\n```\n\"plugins\": [{\n  \"name\": \"semantic_cache\",\n  \"config\": { \"dimension\": 1,\n              \"vector_store_namespace\": \"BifrostBench\",\n              \"default_cache_key\": \"support-cache\",\n              \"ttl\": \"5m\" }\n}]\n```\n\nTwo identical requests, one cache key. The second response:\n\n```\n\"cache_debug\": {\n  \"cache_hit\": true,\n  \"cache_id\": \"1cf8a91b-c115-57bf-97a0-fc821dc4de1e\",\n  \"hit_type\": \"direct\",\n  \"cache_hit_latency\": 0\n}\n```\n\nSame `created`\n\ntimestamp as the first response — it was replayed, not re-fetched. Zero provider call, zero tokens. (Practical note: this needed Redis Stack with the RediSearch module; plain Redis lacks the `FT.*`\n\ncommands the index wants.)\n\n[Virtual keys](https://docs.getbifrost.ai/features/governance/virtual-keys) are the governance primitive: per-team keys carrying model allowlists, budgets, and rate limits. Declared in config for the support team:\n\n```\n\"governance\": {\n  \"virtual_keys\": [{\n    \"id\": \"vk-support-team\",\n    \"value\": \"sk-bf-support-team\",\n    \"provider_configs\": [{\n      \"provider\": \"openai\",\n      \"allowed_models\": [\"support-chat\"], \"key_ids\": [\"*\"]\n    }]\n  }]\n}\n```\n\nThe allowed request routes normally. A request for `premium-model`\n\nwith the same key:\n\n```\n\"Model 'premium-model' is not allowed for this virtual key\"\n```\n\nDenied at the gateway before any provider saw it. The same key machinery carries [budgets and rate limits](https://docs.getbifrost.ai/features/governance/budget-and-limits) — the mechanism that ends the \"who spent $800 on Opus last night\" incident review. Screenshot of the key in the governance UI:\n\nBifrost exports Prometheus metrics natively and logs every request with routing context — provider chosen, fallback index, cache behavior, token counts, latency split into gateway vs upstream time. That last distinction matters: when a provider slows down, you see `upstream_latency`\n\ngrow while gateway overhead stays flat, so you know whose pager to page. See the [observability docs](https://docs.getbifrost.ai/features/observability/default). And for agent traffic, [MCP](https://docs.getbifrost.ai/mcp/overview) is a first-class surface: the gateway brokers tool calls with explicit execution (no auto-execution unless you opt in).\n\nI hand-rolled a minimal MCP server (one tool, JSON-RPC over HTTP) and registered it as a client. The client list reported `state: healthy`\n\nwith `get_time`\n\ndiscovered. Execution went through the gateway explicitly:\n\n```\nPOST /v1/mcp/tool/execute  {\"function\":{\"name\":\"benchtools-get_time\",\"arguments\":\"{}\"}}\n→ {\"role\":\"tool\",\"content\":\"2026-08-27T07:48:21Z\"}\n```\n\nTwo security properties surfaced unprompted: tool names are namespaced per client (`benchtools-get_time`\n\n) to prevent collisions between servers, and execution without permission fails closed (\"tool is not available or not permitted\"). Agent traffic gets the same governance as chat traffic.\n\nRemaining clients migrate one at a time — each is a base-URL change with the gateway's request log as your audit trail. The Logs view after a few requests:\n\nSame 60-request traffic, now through Bifrost with the cache on and the fallback wired:\n\n```\nvia Bifrost: 60 ok / 0 fail, 32 cache hits, 4,363 tokens billed\n```\n\nAt an illustrative $0.0025 per 1K tokens:\n\n| legacy | via Bifrost | |\n|---|---|---|\n| billable tokens | 9,335 | 4,363 |\n| cost | $0.0233 | $0.0109 |\n| cache hits | 0 | 32 |\n| failed requests (provider kill) | 26 | 0 |\n| savings | — | 53% |\n\nThe savings came entirely from replayed cache hits — no provider call, no tokens. On a support workload that repeats questions daily, that ratio compounds. The availability delta speaks for itself: 0 failures through a provider kill, against 26 in the legacy run.\n\nMigrating to an enterprise AI gateway is not a rewrite. It is a sequence of small, reversible moves — deploy beside, point one client, add fallback, enable cache, issue keys, wire observability, cut over. Measured on the rebuilt stack: 53% cost reduction on cacheable traffic, zero failed requests through a provider kill, and governance the legacy stack never had. The migration risk is low; the legacy risk is already on your pager.", "url": "https://wpnews.pro/news/migrating-legacy-llm-infrastructure-to-an-ai-gateway", "canonical_source": "https://dev.to/copyleftdev/migrating-legacy-llm-infrastructure-to-an-ai-gateway-27hl", "published_at": "2026-09-01 13:33:51+00:00", "updated_at": "2026-09-01 13:53:37.532374+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-products"], "entities": ["Bifrost", "Maxim AI", "OpenAI", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/migrating-legacy-llm-infrastructure-to-an-ai-gateway", "markdown": "https://wpnews.pro/news/migrating-legacy-llm-infrastructure-to-an-ai-gateway.md", "text": "https://wpnews.pro/news/migrating-legacy-llm-infrastructure-to-an-ai-gateway.txt", "jsonld": "https://wpnews.pro/news/migrating-legacy-llm-infrastructure-to-an-ai-gateway.jsonld"}}