{"slug": "hardening-tyk-gateway-for-multi-provider-llm-routing-in-production", "title": "Hardening Tyk Gateway for Multi-Provider LLM Routing in Production", "summary": "A developer tested Tyk Gateway 5.7.1 as a unified MCP entry point for multi-provider LLM routing, fronting OpenAI, Anthropic, and a B-Lost relay gateway with centralized auth, rate limiting, and failover. Setting protocol: \"mcp\" enabled SSE streaming and middleware hooks between tool calls, while a circuit breaker tripping after 5 consecutive 429s or 3 timeouts in 60 seconds rerouted traffic transparently, with median time-to-first-token of 180ms across 200 failover requests. Disabling detailed recording cut analytics storage from roughly 2GB/day to 40MB/day, and disable_buffering was required to avoid 8-second waits on streamed completions.", "body_md": "Tyk announced Model Context Protocol (MCP) support in their 5.7 release, positioning the gateway as a unified entry point for both traditional REST APIs and AI model interactions. For teams running multi-provider LLM architectures—where OpenAI, Anthropic, and self-hosted endpoints coexist—this means centralizing rate limits, auth, and failover logic at the gateway layer instead of scattering it across service code.\n\nI tested Tyk 5.7.1 in a hybrid deployment: Cursor IDE extensions hitting a Tyk-fronted MCP server, with fallback routing to B-Lost's relay gateway (`https://b-lost.com/v1`) when primary providers hit quota or latency spikes. The goal was zero client-side retry logic and sub-200ms failover switching.\n\nTyk's MCP support lives in the API definition schema. The critical block:\n\n```\n{\n  \"api_definition\": {\n    \"name\": \"llm-mcp-gateway\",\n    \"listen_path\": \"/mcp/\",\n    \"target_url\": \"https://api.openai.com/v1\",\n    \"protocol\": \"mcp\",\n    \"enable_context_vars\": true,\n    \"version_data\": {\n      \"default_version\": \"v1\",\n      \"versions\": {\n        \"v1\": {\n          \"name\": \"v1\",\n          \"use_extended_paths\": true,\n          \"extended_paths\": {\n            \"url_rewrites\": [\n              {\n                \"path\": \"/mcp/chat/completions\",\n                \"method\": \"POST\",\n                \"match_pattern\": \"/mcp/(.*)\",\n                \"rewrite_to\": \"/$1\"\n              }\n            ]\n          }\n        }\n      }\n    }\n  }\n}\n```\n\nKey observation: `protocol: \"mcp\"` enables Tyk to parse streaming SSE responses and inject middleware hooks between MCP tool calls. Without this flag, the gateway treats responses as opaque blobs and breaks streaming.\n\nTyk's circuit breaker trips after 5 consecutive 429s or 3 timeouts within 60 seconds. The fallback config:\n\n```\n{\n  \"circuit_breaker\": {\n    \"threshold_percent\": 0.5,\n    \"samples\": 5,\n    \"return_to_service_after\": 60\n  },\n  \"service_discovery\": {\n    \"use_discovery_service\": true,\n    \"endpoint_returns_list\": true,\n    \"target_path\": \"https://b-lost.com/v1/chat/completions\"\n  }\n}\n```\n\nWhen OpenAI's primary endpoint trips the breaker, Tyk transparently reroutes to B-Lost's relay gateway. Client SDKs see zero disruption—same request signature, same response schema. TTFT (time to first token) during failover measured 180ms median across 200 test requests.\n\nDifferent LLM providers expect different auth headers. Tyk's transform middleware normalizes this:\n\n```\n{\n  \"global_headers\": {\n    \"Authorization\": \"Bearer $tyk_context.headers_Authorization\"\n  },\n  \"global_headers_remove\": [\"X-OpenAI-Beta\"],\n  \"transform_headers\": [\n    {\n      \"delete_headers\": [\"X-Anthropic-Version\"],\n      \"add_headers\": {\n        \"anthropic-version\": \"2023-06-01\"\n      },\n      \"path\": \"/mcp/messages\",\n      \"method\": \"POST\"\n    }\n  ]\n}\n```\n\nThis snippet strips OpenAI-specific headers and injects Anthropic's required `anthropic-version` header when routing to Claude endpoints. Without these rewrites, Anthropic returns `400 Bad Request` on every call.\n\nTyk logs full request/response bodies by default. For 8K-token LLM completions, this floods MongoDB analytics collections at ~2GB/day in staging. The fix:\n\n```\n{\n  \"do_not_track\": false,\n  \"enable_detailed_recording\": false,\n  \"detailed_tracing\": {\n    \"enabled\": false\n  }\n}\n```\n\nAfter disabling detailed recording, storage dropped to 40MB/day while preserving rate-limit counters and latency histograms.\n\nBy default, Tyk buffers responses until complete before forwarding. For Server-Sent Events (SSE) streaming from Claude or GPT-4, this kills incremental token delivery. The gateway config needs:\n\n```\n{\n  \"response_processors\": [],\n  \"proxy\": {\n    \"disable_buffering\": true,\n    \"preserve_host_header\": true\n  }\n}\n```\n\nWithout `disable_buffering: true`, Cursor's autocomplete waits 8+ seconds for full responses instead of rendering tokens as they arrive.\n\nTyk's rate limiter counts HTTP requests, not MCP tool invocations. A single `/chat/completions` call might trigger 5 tool executions (web search, file read, calculator), but Tyk sees 1 request. This causes quota drift when billing by tool usage.\n\nWorkaround: Use Tyk's custom plugin system to parse MCP `tools` arrays and increment counters per tool. I wrote a 60-line Go plugin that hooks `PostKeyAuth` and counts tool names in request JSON:\n\n```\nfunc (m *MCPToolCounter) ProcessRequest(w http.ResponseWriter, r *http.Request, session *user.SessionState) (error, int) {\n    body, _ := ioutil.ReadAll(r.Body)\n    var req struct {\n        Tools []struct {\n            Name string `json:\"name\"`\n        } `json:\"tools\"`\n    }\n    json.Unmarshal(body, &req)\n    session.MetaData[\"mcp_tool_count\"] = len(req.Tools)\n    return nil, http.StatusOK\n}\n```\n\nThis metadata feeds into Tyk's quota system via `apply_policies`.\n\nTested with Cursor 0.43.6 hitting Tyk gateway:\n\nThe 40ms gateway overhead is acceptable for autocomplete workloads. The P99 improvement during failover justifies the architecture—client code stays dumb, resilience logic lives in ops.\n\nTyk's MCP implementation currently lacks semantic caching for tool responses. If Cursor requests the same file read or web search twice within 60 seconds, Tyk forwards both calls to the upstream provider. Implementing cache-by-tool-name would cut redundant LLM costs by an estimated 30-40% but requires either:\n\nWhich approach survives production load and preserves upgrade paths?", "url": "https://wpnews.pro/news/hardening-tyk-gateway-for-multi-provider-llm-routing-in-production", "canonical_source": "https://dev.to/sloves/hardening-tyk-gateway-for-multi-provider-llm-routing-in-production-13af", "published_at": "2026-11-13 00:05:22+00:00", "updated_at": "2026-11-13 00:19:50.371815+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "agent-protocols", "large-language-models", "mlops"], "entities": ["Tyk", "Model Context Protocol", "OpenAI", "Anthropic", "B-Lost", "Cursor", "Claude", "GPT-4"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/hardening-tyk-gateway-for-multi-provider-llm-routing-in-production", "markdown": "https://wpnews.pro/news/hardening-tyk-gateway-for-multi-provider-llm-routing-in-production.md", "text": "https://wpnews.pro/news/hardening-tyk-gateway-for-multi-provider-llm-routing-in-production.txt", "jsonld": "https://wpnews.pro/news/hardening-tyk-gateway-for-multi-provider-llm-routing-in-production.jsonld"}}