# Route Requests Across GPT-5.6, Claude, and Gemini with a Unified LLM Gateway > Source: > Published: 2026-07-18 07:41:35+00:00 # Route Requests Across GPT-5.6, Claude, and Gemini with a Unified LLM Gateway Stand up a local LiteLLM proxy that load-balances and fails over across OpenAI, Anthropic, and Gemini behind one OpenAI-compatible endpoint. [Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein) ## What you'll build A local LiteLLM proxy that exposes a single OpenAI-compatible `/v1/chat/completions` endpoint. Behind that one endpoint, it load-balances requests across OpenAI, Anthropic, and Gemini, automatically retries on a different provider if one fails, and reports per-request cost. Your app code never has to know which provider actually served the request. One naming note up front: this tutorial's title uses "GPT-5.6" as shorthand for "whichever GPT model is current when you read this." No such model has shipped. The working config below uses `gpt-4o` , a current OpenAI production model at the time of writing, so every command here actually runs. LiteLLM doesn't validate model strings itself, it just passes whatever you put in `model:` straight to the provider, so swapping in a newer OpenAI model later means changing one line and nothing else. ## Prerequisites - Python 3.9 or newer ( `python3 --version` ) - API keys for the providers you want behind the gateway: an OpenAI key, an Anthropic key, and a Google AI Studio (Gemini) key - macOS, Linux, or WSL2 on Windows. No compiled dependencies here, so Apple Silicon vs Intel doesn't matter - Basic comfort with YAML and curl ## Step 1: Install LiteLLM Use a virtual environment so this doesn't collide with other Python projects. ``` python3 -m venv venv source venv/bin/activate pip install 'litellm[proxy]' ``` Confirm it installed: ``` pip show litellm ``` You should see a `Version:` line. That's enough to confirm the install worked, don't rely on a `--version` flag on the `litellm` CLI itself, since not every build supports it consistently. If you'd rather skip the local Python environment entirely, the official Docker image works the same way: ``` docker pull ghcr.io/berriai/litellm:main-latest ``` ## Step 2: Set your provider credentials LiteLLM reads provider keys from environment variables. Export them in the shell where you'll run the proxy: ``` export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." export GEMINI_API_KEY="AIza..." export LITELLM_MASTER_KEY="sk-gateway-1234" ``` `LITELLM_MASTER_KEY` isn't a provider key, it's the credential clients will use to authenticate to *your* gateway. Pick a random string, don't reuse a provider key here. Don't put any of these in a committed file. If you want persistence across shells, put the exports in a `.env` file that's in your `.gitignore` , or use your OS keychain. ## Step 3: Write the gateway config Create `config.yaml` : ``` model_list: - model_name: smart-router litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY - model_name: smart-router litellm_params: model: anthropic/claude-3-5-sonnet-20241022 api_key: os.environ/ANTHROPIC_API_KEY - model_name: smart-router litellm_params: model: gemini/gemini-2.0-flash api_key: os.environ/GEMINI_API_KEY router_settings: routing_strategy: simple-shuffle num_retries: 2 timeout: 30 allowed_fails: 2 cooldown_time: 30 litellm_settings: drop_params: true general_settings: master_key: os.environ/LITELLM_MASTER_KEY ``` A few things worth understanding here, not just copying: - Giving all three deployments the same `model_name` (`smart-router` ) is what makes them one logical group. Clients call`smart-router` , LiteLLM picks a deployment from the group and load-balances across them. `routing_strategy: simple-shuffle` is the default and requires no external infrastructure. It picks randomly (weighted if you set`weight` per deployment) among healthy deployments. Don't reach for`latency-based-routing` yet, it needs a Redis backend to store rolling latency stats across requests, more on that in Next Steps.`num_retries` and`allowed_fails` are your failover knobs. If the OpenAI deployment throws an error, LiteLLM retries against Claude or Gemini in the same group instead of just failing the request.`cooldown_time` temporarily benches a deployment that's been erroring, so you're not hammering a provider that's down.`drop_params: true` strips provider-unsupported request params instead of erroring, useful since not every provider accepts every OpenAI parameter (like`logprobs` on some Gemini models). If you'd rather use a smaller, cheaper OpenAI model for this exercise, swap `openai/gpt-4o` for `openai/gpt-4o-mini` , both are current and work identically here. ## Step 4: Start the proxy ``` litellm --config config.yaml --port 4000 ``` You should see log lines confirming each deployment loaded, plus a line saying the proxy is running on `http://0.0.0.0:4000` . Leave this running in its own terminal tab. Docker equivalent, mounting your config in: ``` docker run -p 4000:4000 \ -e OPENAI_API_KEY -e ANTHROPIC_API_KEY -e GEMINI_API_KEY -e LITELLM_MASTER_KEY \ -v $(pwd)/config.yaml:/app/config.yaml \ ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000 ``` ## Step 5: Call it like any OpenAI endpoint Raw curl: ``` curl -i http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer sk-gateway-1234" \ -H "Content-Type: application/json" \ -d '{ "model": "smart-router", "messages": [{"role": "user", "content": "Say hello in exactly 5 words."}] }' ``` Or from Python, using the official `openai` SDK unchanged except for `base_url` : ``` python from openai import OpenAI client = OpenAI( api_key="sk-gateway-1234", base_url="http://localhost:4000/v1", ) resp = client.chat.completions.create( model="smart-router", messages=[{"role": "user", "content": "Explain load balancing in one sentence."}], ) print(resp.choices[0].message.content) ``` Note the `/v1` suffix on `base_url` . LiteLLM happens to also expose some routes at the root, but the OpenAI SDK (and most other OpenAI-compatible clients) assumes the standard `/v1` prefix and will build request paths accordingly. Pointing at root works by accident on LiteLLM specifically; pointing at `/v1` works everywhere, so just do that. Run this a handful of times and you'll notice the model actually answering isn't always the same provider (check the `model` field in the JSON response to confirm, LiteLLM rewrites it to the real underlying model it hit). ## Step 6: Check cost per request Run the curl command with `-i` (as above) and look at the response headers. LiteLLM adds `x-litellm-response-cost` , the dollar cost of that specific call, computed from the provider's published token pricing: ``` curl -i http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer sk-gateway-1234" \ -H "Content-Type: application/json" \ -d '{"model": "smart-router", "messages": [{"role": "user", "content": "hi"}]}' \ | grep -i x-litellm ``` For latency, `time curl ...` is enough to eyeball round-trip time locally. For real dashboards across many requests, you want Prometheus, covered in Next Steps. ## Verify it works - The proxy starts without errors and logs all three deployments under `smart-router` . - A curl request returns a normal OpenAI-shaped chat completion ( `choices[0].message.content` ). - The `x-litellm-response-cost` header is present and non-zero. - Temporarily set one provider's env var to an invalid value (e.g. `export OPENAI_API_KEY=invalid` ), restart the proxy, and send several requests. Some should still succeed, served by Claude or Gemini, proving the failover path actually works instead of just existing on paper. ## Troubleshooting : the`AuthenticationError: No models match model_name` or similar`model` field in your request body has to exactly match a`model_name` in`config.yaml` . Typos here are the most common failure.**401 from the gateway itself**: you forgot`Bearer` in the`Authorization` header, or the token doesn't match`LITELLM_MASTER_KEY` . Note the env var is only read at proxy startup, restart after changing it.**401/403 bubbling up from a specific provider**: usually a stale or mistyped key. Double-check the exact env var name LiteLLM expects per provider (`GEMINI_API_KEY` , not`GOOGLE_API_KEY` , for the`gemini/` prefix) and confirm you exported it in the same shell you launched the proxy from.**All requests failing after one provider goes down**: check`cooldown_time` and`allowed_fails` , if they're too aggressive the whole group can get benched. Also confirm you actually have more than one deployment under the same`model_name` , load balancing across a group of one does nothing. ## Next steps **Latency-based routing**: switch`routing_strategy` to`latency-based-routing` once you're ready, but it needs a Redis instance to store rolling per-deployment latency across requests and processes (a single in-memory process can't do this reliably at any real traffic level). Add`redis_host` ,`redis_port` , and`redis_password` under`router_settings` and point them at a Redis container or managed instance.**Persistent cost/spend tracking and the built-in UI**: point LiteLLM at a Postgres database via`DATABASE_URL` to unlock the`/ui` admin dashboard, virtual API keys, and per-team budgets, instead of reading cost off response headers one call at a time.**Prometheus metrics**: add`prometheus` to`litellm_settings.success_callback` to get a`/metrics` endpoint with latency histograms and request counts per model, then wire it into Grafana.**Production deployment**: run this behind a real reverse proxy with TLS, and read LiteLLM's docs on rate limiting and budget alerts before you put real traffic behind it. [Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)ยท Dev Tools Editor Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop. ## Discussion 0 No comments yet Be the first to weigh in.