{"slug": "route-requests-across-gpt-5-6-claude-and-gemini-with-a-unified-llm-gateway", "title": "Route Requests Across GPT-5.6, Claude, and Gemini with a Unified LLM Gateway", "summary": "LiteLLM, an open-source proxy, enables developers to route requests across OpenAI, Anthropic, and Gemini models through a single OpenAI-compatible endpoint, providing load balancing, failover, and cost tracking. The tool simplifies multi-provider LLM integration by abstracting provider-specific APIs behind a unified gateway, reducing code complexity and improving reliability. This approach allows applications to switch or combine models without changing application logic, addressing the growing need for flexible and resilient AI infrastructure.", "body_md": "# Route Requests Across GPT-5.6, Claude, and Gemini with a Unified LLM Gateway\n\nStand up a local LiteLLM proxy that load-balances and fails over across OpenAI, Anthropic, and Gemini behind one OpenAI-compatible endpoint.\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)\n\n## What you'll build\n\nA local LiteLLM proxy that exposes a single OpenAI-compatible `/v1/chat/completions`\n\nendpoint. Behind that one endpoint, it load-balances requests across OpenAI, Anthropic, and Gemini, automatically retries on a different provider if one fails, and reports per-request cost. Your app code never has to know which provider actually served the request.\n\nOne naming note up front: this tutorial's title uses \"GPT-5.6\" as shorthand for \"whichever GPT model is current when you read this.\" No such model has shipped. The working config below uses `gpt-4o`\n\n, a current OpenAI production model at the time of writing, so every command here actually runs. LiteLLM doesn't validate model strings itself, it just passes whatever you put in `model:`\n\nstraight to the provider, so swapping in a newer OpenAI model later means changing one line and nothing else.\n\n## Prerequisites\n\n- Python 3.9 or newer (\n`python3 --version`\n\n) - API keys for the providers you want behind the gateway: an OpenAI key, an Anthropic key, and a Google AI Studio (Gemini) key\n- macOS, Linux, or WSL2 on Windows. No compiled dependencies here, so Apple Silicon vs Intel doesn't matter\n- Basic comfort with YAML and curl\n\n## Step 1: Install LiteLLM\n\nUse a virtual environment so this doesn't collide with other Python projects.\n\n```\npython3 -m venv venv\nsource venv/bin/activate\npip install 'litellm[proxy]'\n```\n\nConfirm it installed:\n\n```\npip show litellm\n```\n\nYou should see a `Version:`\n\nline. That's enough to confirm the install worked, don't rely on a `--version`\n\nflag on the `litellm`\n\nCLI itself, since not every build supports it consistently.\n\nIf you'd rather skip the local Python environment entirely, the official Docker image works the same way:\n\n```\ndocker pull ghcr.io/berriai/litellm:main-latest\n```\n\n## Step 2: Set your provider credentials\n\nLiteLLM reads provider keys from environment variables. Export them in the shell where you'll run the proxy:\n\n```\nexport OPENAI_API_KEY=\"sk-...\"\nexport ANTHROPIC_API_KEY=\"sk-ant-...\"\nexport GEMINI_API_KEY=\"AIza...\"\nexport LITELLM_MASTER_KEY=\"sk-gateway-1234\"\n```\n\n`LITELLM_MASTER_KEY`\n\nisn't a provider key, it's the credential clients will use to authenticate to *your* gateway. Pick a random string, don't reuse a provider key here.\n\nDon't put any of these in a committed file. If you want persistence across shells, put the exports in a `.env`\n\nfile that's in your `.gitignore`\n\n, or use your OS keychain.\n\n## Step 3: Write the gateway config\n\nCreate `config.yaml`\n\n:\n\n```\nmodel_list:\n  - model_name: smart-router\n    litellm_params:\n      model: openai/gpt-4o\n      api_key: os.environ/OPENAI_API_KEY\n\n  - model_name: smart-router\n    litellm_params:\n      model: anthropic/claude-3-5-sonnet-20241022\n      api_key: os.environ/ANTHROPIC_API_KEY\n\n  - model_name: smart-router\n    litellm_params:\n      model: gemini/gemini-2.0-flash\n      api_key: os.environ/GEMINI_API_KEY\n\nrouter_settings:\n  routing_strategy: simple-shuffle\n  num_retries: 2\n  timeout: 30\n  allowed_fails: 2\n  cooldown_time: 30\n\nlitellm_settings:\n  drop_params: true\n\ngeneral_settings:\n  master_key: os.environ/LITELLM_MASTER_KEY\n```\n\nA few things worth understanding here, not just copying:\n\n- Giving all three deployments the same\n`model_name`\n\n(`smart-router`\n\n) is what makes them one logical group. Clients call`smart-router`\n\n, LiteLLM picks a deployment from the group and load-balances across them. `routing_strategy: simple-shuffle`\n\nis the default and requires no external infrastructure. It picks randomly (weighted if you set`weight`\n\nper deployment) among healthy deployments. Don't reach for`latency-based-routing`\n\nyet, it needs a Redis backend to store rolling latency stats across requests, more on that in Next Steps.`num_retries`\n\nand`allowed_fails`\n\nare your failover knobs. If the OpenAI deployment throws an error, LiteLLM retries against Claude or Gemini in the same group instead of just failing the request.`cooldown_time`\n\ntemporarily benches a deployment that's been erroring, so you're not hammering a provider that's down.`drop_params: true`\n\nstrips provider-unsupported request params instead of erroring, useful since not every provider accepts every OpenAI parameter (like`logprobs`\n\non some Gemini models).\n\nIf you'd rather use a smaller, cheaper OpenAI model for this exercise, swap `openai/gpt-4o`\n\nfor `openai/gpt-4o-mini`\n\n, both are current and work identically here.\n\n## Step 4: Start the proxy\n\n```\nlitellm --config config.yaml --port 4000\n```\n\nYou should see log lines confirming each deployment loaded, plus a line saying the proxy is running on `http://0.0.0.0:4000`\n\n. Leave this running in its own terminal tab.\n\nDocker equivalent, mounting your config in:\n\n```\ndocker run -p 4000:4000 \\\n  -e OPENAI_API_KEY -e ANTHROPIC_API_KEY -e GEMINI_API_KEY -e LITELLM_MASTER_KEY \\\n  -v $(pwd)/config.yaml:/app/config.yaml \\\n  ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000\n```\n\n## Step 5: Call it like any OpenAI endpoint\n\nRaw curl:\n\n```\ncurl -i http://localhost:4000/v1/chat/completions \\\n  -H \"Authorization: Bearer sk-gateway-1234\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"smart-router\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Say hello in exactly 5 words.\"}]\n  }'\n```\n\nOr from Python, using the official `openai`\n\nSDK unchanged except for `base_url`\n\n:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    api_key=\"sk-gateway-1234\",\n    base_url=\"http://localhost:4000/v1\",\n)\n\nresp = client.chat.completions.create(\n    model=\"smart-router\",\n    messages=[{\"role\": \"user\", \"content\": \"Explain load balancing in one sentence.\"}],\n)\nprint(resp.choices[0].message.content)\n```\n\nNote the `/v1`\n\nsuffix on `base_url`\n\n. LiteLLM happens to also expose some routes at the root, but the OpenAI SDK (and most other OpenAI-compatible clients) assumes the standard `/v1`\n\nprefix and will build request paths accordingly. Pointing at root works by accident on LiteLLM specifically; pointing at `/v1`\n\nworks everywhere, so just do that.\n\nRun this a handful of times and you'll notice the model actually answering isn't always the same provider (check the `model`\n\nfield in the JSON response to confirm, LiteLLM rewrites it to the real underlying model it hit).\n\n## Step 6: Check cost per request\n\nRun the curl command with `-i`\n\n(as above) and look at the response headers. LiteLLM adds `x-litellm-response-cost`\n\n, the dollar cost of that specific call, computed from the provider's published token pricing:\n\n```\ncurl -i http://localhost:4000/v1/chat/completions \\\n  -H \"Authorization: Bearer sk-gateway-1234\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"smart-router\", \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}]}' \\\n  | grep -i x-litellm\n```\n\nFor latency, `time curl ...`\n\nis enough to eyeball round-trip time locally. For real dashboards across many requests, you want Prometheus, covered in Next Steps.\n\n## Verify it works\n\n- The proxy starts without errors and logs all three deployments under\n`smart-router`\n\n. - A curl request returns a normal OpenAI-shaped chat completion (\n`choices[0].message.content`\n\n). - The\n`x-litellm-response-cost`\n\nheader is present and non-zero. - Temporarily set one provider's env var to an invalid value (e.g.\n`export OPENAI_API_KEY=invalid`\n\n), restart the proxy, and send several requests. Some should still succeed, served by Claude or Gemini, proving the failover path actually works instead of just existing on paper.\n\n## Troubleshooting\n\n: the`AuthenticationError: No models match model_name`\n\nor similar`model`\n\nfield in your request body has to exactly match a`model_name`\n\nin`config.yaml`\n\n. Typos here are the most common failure.**401 from the gateway itself**: you forgot`Bearer`\n\nin the`Authorization`\n\nheader, or the token doesn't match`LITELLM_MASTER_KEY`\n\n. Note the env var is only read at proxy startup, restart after changing it.**401/403 bubbling up from a specific provider**: usually a stale or mistyped key. Double-check the exact env var name LiteLLM expects per provider (`GEMINI_API_KEY`\n\n, not`GOOGLE_API_KEY`\n\n, for the`gemini/`\n\nprefix) and confirm you exported it in the same shell you launched the proxy from.**All requests failing after one provider goes down**: check`cooldown_time`\n\nand`allowed_fails`\n\n, if they're too aggressive the whole group can get benched. Also confirm you actually have more than one deployment under the same`model_name`\n\n, load balancing across a group of one does nothing.\n\n## Next steps\n\n**Latency-based routing**: switch`routing_strategy`\n\nto`latency-based-routing`\n\nonce you're ready, but it needs a Redis instance to store rolling per-deployment latency across requests and processes (a single in-memory process can't do this reliably at any real traffic level). Add`redis_host`\n\n,`redis_port`\n\n, and`redis_password`\n\nunder`router_settings`\n\nand point them at a Redis container or managed instance.**Persistent cost/spend tracking and the built-in UI**: point LiteLLM at a Postgres database via`DATABASE_URL`\n\nto unlock the`/ui`\n\nadmin dashboard, virtual API keys, and per-team budgets, instead of reading cost off response headers one call at a time.**Prometheus metrics**: add`prometheus`\n\nto`litellm_settings.success_callback`\n\nto get a`/metrics`\n\nendpoint with latency histograms and request counts per model, then wire it into Grafana.**Production deployment**: run this behind a real reverse proxy with TLS, and read LiteLLM's docs on rate limiting and budget alerts before you put real traffic behind it.\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor\n\nRachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/route-requests-across-gpt-5-6-claude-and-gemini-with-a-unified-llm-gateway", "canonical_source": "https://sourcefeed.dev/a/route-requests-across-gpt-56-claude-and-gemini-with-a-unified-llm-gateway", "published_at": "2026-07-18 07:41:35+00:00", "updated_at": "2026-07-18 07:54:23.916952+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["LiteLLM", "OpenAI", "Anthropic", "Gemini", "Rachel Goldstein"], "alternates": {"html": "https://wpnews.pro/news/route-requests-across-gpt-5-6-claude-and-gemini-with-a-unified-llm-gateway", "markdown": "https://wpnews.pro/news/route-requests-across-gpt-5-6-claude-and-gemini-with-a-unified-llm-gateway.md", "text": "https://wpnews.pro/news/route-requests-across-gpt-5-6-claude-and-gemini-with-a-unified-llm-gateway.txt", "jsonld": "https://wpnews.pro/news/route-requests-across-gpt-5-6-claude-and-gemini-with-a-unified-llm-gateway.jsonld"}}