Hardening Tyk Gateway for Multi-Provider LLM Routing in Production

A developer tested Tyk Gateway 5.7.1 as a unified MCP entry point for multi-provider LLM routing, fronting OpenAI, Anthropic, and a B-Lost relay gateway with centralized auth, rate limiting, and failover. Setting protocol: "mcp" enabled SSE streaming and middleware hooks between tool calls, while a circuit breaker tripping after 5 consecutive 429s or 3 timeouts in 60 seconds rerouted traffic transparently, with median time-to-first-token of 180ms across 200 failover requests. Disabling detailed recording cut analytics storage from roughly 2GB/day to 40MB/day, and disable_buffering was required to avoid 8-second waits on streamed completions.

Tyk announced Model Context Protocol MCP support in their 5.7 release, positioning the gateway as a unified entry point for both traditional REST APIs and AI model interactions. For teams running multi-provider LLM architectures—where OpenAI, Anthropic, and self-hosted endpoints coexist—this means centralizing rate limits, auth, and failover logic at the gateway layer instead of scattering it across service code. I tested Tyk 5.7.1 in a hybrid deployment: Cursor IDE extensions hitting a Tyk-fronted MCP server, with fallback routing to B-Lost's relay gateway https://b-lost.com/v1 when primary providers hit quota or latency spikes. The goal was zero client-side retry logic and sub-200ms failover switching. Tyk's MCP support lives in the API definition schema. The critical block: { "api definition": { "name": "llm-mcp-gateway", "listen path": "/mcp/", "target url": "https://api.openai.com/v1", "protocol": "mcp", "enable context vars": true, "version data": { "default version": "v1", "versions": { "v1": { "name": "v1", "use extended paths": true, "extended paths": { "url rewrites": { "path": "/mcp/chat/completions", "method": "POST", "match pattern": "/mcp/ . ", "rewrite to": "/$1" } } } } } } } Key observation: protocol: "mcp" enables Tyk to parse streaming SSE responses and inject middleware hooks between MCP tool calls. Without this flag, the gateway treats responses as opaque blobs and breaks streaming. Tyk's circuit breaker trips after 5 consecutive 429s or 3 timeouts within 60 seconds. The fallback config: { "circuit breaker": { "threshold percent": 0.5, "samples": 5, "return to service after": 60 }, "service discovery": { "use discovery service": true, "endpoint returns list": true, "target path": "https://b-lost.com/v1/chat/completions" } } When OpenAI's primary endpoint trips the breaker, Tyk transparently reroutes to B-Lost's relay gateway. Client SDKs see zero disruption—same request signature, same response schema. TTFT time to first token during failover measured 180ms median across 200 test requests. Different LLM providers expect different auth headers. Tyk's transform middleware normalizes this: { "global headers": { "Authorization": "Bearer $tyk context.headers Authorization" }, "global headers remove": "X-OpenAI-Beta" , "transform headers": { "delete headers": "X-Anthropic-Version" , "add headers": { "anthropic-version": "2023-06-01" }, "path": "/mcp/messages", "method": "POST" } } This snippet strips OpenAI-specific headers and injects Anthropic's required anthropic-version header when routing to Claude endpoints. Without these rewrites, Anthropic returns 400 Bad Request on every call. Tyk logs full request/response bodies by default. For 8K-token LLM completions, this floods MongoDB analytics collections at ~2GB/day in staging. The fix: { "do not track": false, "enable detailed recording": false, "detailed tracing": { "enabled": false } } After disabling detailed recording, storage dropped to 40MB/day while preserving rate-limit counters and latency histograms. By default, Tyk buffers responses until complete before forwarding. For Server-Sent Events SSE streaming from Claude or GPT-4, this kills incremental token delivery. The gateway config needs: { "response processors": , "proxy": { "disable buffering": true, "preserve host header": true } } Without disable buffering: true , Cursor's autocomplete waits 8+ seconds for full responses instead of rendering tokens as they arrive. Tyk's rate limiter counts HTTP requests, not MCP tool invocations. A single /chat/completions call might trigger 5 tool executions web search, file read, calculator , but Tyk sees 1 request. This causes quota drift when billing by tool usage. Workaround: Use Tyk's custom plugin system to parse MCP tools arrays and increment counters per tool. I wrote a 60-line Go plugin that hooks PostKeyAuth and counts tool names in request JSON: func m MCPToolCounter ProcessRequest w http.ResponseWriter, r http.Request, session user.SessionState error, int { body, := ioutil.ReadAll r.Body var req struct { Tools struct { Name string json:"name" } json:"tools" } json.Unmarshal body, &req session.MetaData "mcp tool count" = len req.Tools return nil, http.StatusOK } This metadata feeds into Tyk's quota system via apply policies . Tested with Cursor 0.43.6 hitting Tyk gateway: The 40ms gateway overhead is acceptable for autocomplete workloads. The P99 improvement during failover justifies the architecture—client code stays dumb, resilience logic lives in ops. Tyk's MCP implementation currently lacks semantic caching for tool responses. If Cursor requests the same file read or web search twice within 60 seconds, Tyk forwards both calls to the upstream provider. Implementing cache-by-tool-name would cut redundant LLM costs by an estimated 30-40% but requires either: Which approach survives production load and preserves upgrade paths?