Hardening Tyk Gateway for Multi-Provider LLM Routing in Production

Tyk announced Model Context Protocol (MCP) support in their 5.7 release, positioning the gateway as a unified entry point for both traditional REST APIs and AI model interactions. For teams running multi-provider LLM architectures—where OpenAI, Anthropic, and self-hosted endpoints coexist—this means centralizing rate limits, auth, and failover logic at the gateway layer instead of scattering it across service code.

I tested Tyk 5.7.1 in a hybrid deployment: Cursor IDE extensions hitting a Tyk-fronted MCP server, with fallback routing to B-Lost's relay gateway (https://b-lost.com/v1) when primary providers hit quota or latency spikes. The goal was zero client-side retry logic and sub-200ms failover switching.

Tyk's MCP support lives in the API definition schema. The critical block:

{
  "api_definition": {
    "name": "llm-mcp-gateway",
    "listen_path": "/mcp/",
    "target_url": "https://api.openai.com/v1",
    "protocol": "mcp",
    "enable_context_vars": true,
    "version_data": {
      "default_version": "v1",
      "versions": {
        "v1": {
          "name": "v1",
          "use_extended_paths": true,
          "extended_paths": {
            "url_rewrites": [
              {
                "path": "/mcp/chat/completions",
                "method": "POST",
                "match_pattern": "/mcp/(.*)",
                "rewrite_to": "/$1"
              }
            ]
          }
        }
      }
    }
  }
}

Key observation: protocol: "mcp" enables Tyk to parse streaming SSE responses and inject middleware hooks between MCP tool calls. Without this flag, the gateway treats responses as opaque blobs and breaks streaming.

Tyk's circuit breaker trips after 5 consecutive 429s or 3 timeouts within 60 seconds. The fallback config:

{
  "circuit_breaker": {
    "threshold_percent": 0.5,
    "samples": 5,
    "return_to_service_after": 60
  },
  "service_discovery": {
    "use_discovery_service": true,
    "endpoint_returns_list": true,
    "target_path": "https://b-lost.com/v1/chat/completions"
  }
}

When OpenAI's primary endpoint trips the breaker, Tyk transparently reroutes to B-Lost's relay gateway. Client SDKs see zero disruption—same request signature, same response schema. TTFT (time to first token) during failover measured 180ms median across 200 test requests.

Different LLM providers expect different auth headers. Tyk's transform middleware normalizes this:

{
  "global_headers": {
    "Authorization": "Bearer $tyk_context.headers_Authorization"
  },
  "global_headers_remove": ["X-OpenAI-Beta"],
  "transform_headers": [
    {
      "delete_headers": ["X-Anthropic-Version"],
      "add_headers": {
        "anthropic-version": "2023-06-01"
      },
      "path": "/mcp/messages",
      "method": "POST"
    }
  ]
}

This snippet strips OpenAI-specific headers and injects Anthropic's required anthropic-version header when routing to Claude endpoints. Without these rewrites, Anthropic returns 400 Bad Request on every call.

Tyk logs full request/response bodies by default. For 8K-token LLM completions, this floods MongoDB analytics collections at ~2GB/day in staging. The fix:

{
  "do_not_track": false,
  "enable_detailed_recording": false,
  "detailed_tracing": {
    "enabled": false
  }
}

After disabling detailed recording, storage dropped to 40MB/day while preserving rate-limit counters and latency histograms.

By default, Tyk buffers responses until complete before forwarding. For Server-Sent Events (SSE) streaming from Claude or GPT-4, this kills incremental token delivery. The gateway config needs:

{
  "response_processors": [],
  "proxy": {
    "disable_buffering": true,
    "preserve_host_header": true
  }
}

Without disable_buffering: true, Cursor's autocomplete waits 8+ seconds for full responses instead of rendering tokens as they arrive.

Tyk's rate limiter counts HTTP requests, not MCP tool invocations. A single /chat/completions call might trigger 5 tool executions (web search, file read, calculator), but Tyk sees 1 request. This causes quota drift when billing by tool usage.

Workaround: Use Tyk's custom plugin system to parse MCP tools arrays and increment counters per tool. I wrote a 60-line Go plugin that hooks PostKeyAuth and counts tool names in request JSON:

func (m *MCPToolCounter) ProcessRequest(w http.ResponseWriter, r *http.Request, session *user.SessionState) (error, int) {
    body, _ := ioutil.ReadAll(r.Body)
    var req struct {
        Tools []struct {
            Name string `json:"name"`
        } `json:"tools"`
    }
    json.Unmarshal(body, &req)
    session.MetaData["mcp_tool_count"] = len(req.Tools)
    return nil, http.StatusOK
}

This metadata feeds into Tyk's quota system via apply_policies.

Tested with Cursor 0.43.6 hitting Tyk gateway:

The 40ms gateway overhead is acceptable for autocomplete workloads. The P99 improvement during failover justifies the architecture—client code stays dumb, resilience logic lives in ops.

Tyk's MCP implementation currently lacks semantic caching for tool responses. If Cursor requests the same file read or web search twice within 60 seconds, Tyk forwards both calls to the upstream provider. Implementing cache-by-tool-name would cut redundant LLM costs by an estimated 30-40% but requires either:

Which approach survives production load and preserves upgrade paths?