Route Requests Across GPT-5.6, Claude, and Gemini with a Unified LLM Gateway

LiteLLM, an open-source proxy, enables developers to route requests across OpenAI, Anthropic, and Gemini models through a single OpenAI-compatible endpoint, providing load balancing, failover, and cost tracking. The tool simplifies multi-provider LLM integration by abstracting provider-specific APIs behind a unified gateway, reducing code complexity and improving reliability. This approach allows applications to switch or combine models without changing application logic, addressing the growing need for flexible and resilient AI infrastructure.

Route Requests Across GPT-5.6, Claude, and Gemini with a Unified LLM Gateway Stand up a local LiteLLM proxy that load-balances and fails over across OpenAI, Anthropic, and Gemini behind one OpenAI-compatible endpoint. Rachel Goldstein https://sourcefeed.dev/u/rachel goldstein What you'll build A local LiteLLM proxy that exposes a single OpenAI-compatible /v1/chat/completions endpoint. Behind that one endpoint, it load-balances requests across OpenAI, Anthropic, and Gemini, automatically retries on a different provider if one fails, and reports per-request cost. Your app code never has to know which provider actually served the request. One naming note up front: this tutorial's title uses "GPT-5.6" as shorthand for "whichever GPT model is current when you read this." No such model has shipped. The working config below uses gpt-4o , a current OpenAI production model at the time of writing, so every command here actually runs. LiteLLM doesn't validate model strings itself, it just passes whatever you put in model: straight to the provider, so swapping in a newer OpenAI model later means changing one line and nothing else. Prerequisites - Python 3.9 or newer python3 --version - API keys for the providers you want behind the gateway: an OpenAI key, an Anthropic key, and a Google AI Studio Gemini key - macOS, Linux, or WSL2 on Windows. No compiled dependencies here, so Apple Silicon vs Intel doesn't matter - Basic comfort with YAML and curl Step 1: Install LiteLLM Use a virtual environment so this doesn't collide with other Python projects. python3 -m venv venv source venv/bin/activate pip install 'litellm proxy ' Confirm it installed: pip show litellm You should see a Version: line. That's enough to confirm the install worked, don't rely on a --version flag on the litellm CLI itself, since not every build supports it consistently. If you'd rather skip the local Python environment entirely, the official Docker image works the same way: docker pull ghcr.io/berriai/litellm:main-latest Step 2: Set your provider credentials LiteLLM reads provider keys from environment variables. Export them in the shell where you'll run the proxy: export OPENAI API KEY="sk-..." export ANTHROPIC API KEY="sk-ant-..." export GEMINI API KEY="AIza..." export LITELLM MASTER KEY="sk-gateway-1234" LITELLM MASTER KEY isn't a provider key, it's the credential clients will use to authenticate to your gateway. Pick a random string, don't reuse a provider key here. Don't put any of these in a committed file. If you want persistence across shells, put the exports in a .env file that's in your .gitignore , or use your OS keychain. Step 3: Write the gateway config Create config.yaml : model list: - model name: smart-router litellm params: model: openai/gpt-4o api key: os.environ/OPENAI API KEY - model name: smart-router litellm params: model: anthropic/claude-3-5-sonnet-20241022 api key: os.environ/ANTHROPIC API KEY - model name: smart-router litellm params: model: gemini/gemini-2.0-flash api key: os.environ/GEMINI API KEY router settings: routing strategy: simple-shuffle num retries: 2 timeout: 30 allowed fails: 2 cooldown time: 30 litellm settings: drop params: true general settings: master key: os.environ/LITELLM MASTER KEY A few things worth understanding here, not just copying: - Giving all three deployments the same model name smart-router is what makes them one logical group. Clients call smart-router , LiteLLM picks a deployment from the group and load-balances across them. routing strategy: simple-shuffle is the default and requires no external infrastructure. It picks randomly weighted if you set weight per deployment among healthy deployments. Don't reach for latency-based-routing yet, it needs a Redis backend to store rolling latency stats across requests, more on that in Next Steps. num retries and allowed fails are your failover knobs. If the OpenAI deployment throws an error, LiteLLM retries against Claude or Gemini in the same group instead of just failing the request. cooldown time temporarily benches a deployment that's been erroring, so you're not hammering a provider that's down. drop params: true strips provider-unsupported request params instead of erroring, useful since not every provider accepts every OpenAI parameter like logprobs on some Gemini models . If you'd rather use a smaller, cheaper OpenAI model for this exercise, swap openai/gpt-4o for openai/gpt-4o-mini , both are current and work identically here. Step 4: Start the proxy litellm --config config.yaml --port 4000 You should see log lines confirming each deployment loaded, plus a line saying the proxy is running on http://0.0.0.0:4000 . Leave this running in its own terminal tab. Docker equivalent, mounting your config in: docker run -p 4000:4000 \ -e OPENAI API KEY -e ANTHROPIC API KEY -e GEMINI API KEY -e LITELLM MASTER KEY \ -v $ pwd /config.yaml:/app/config.yaml \ ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000 Step 5: Call it like any OpenAI endpoint Raw curl: curl -i http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer sk-gateway-1234" \ -H "Content-Type: application/json" \ -d '{ "model": "smart-router", "messages": {"role": "user", "content": "Say hello in exactly 5 words."} }' Or from Python, using the official openai SDK unchanged except for base url : python from openai import OpenAI client = OpenAI api key="sk-gateway-1234", base url="http://localhost:4000/v1", resp = client.chat.completions.create model="smart-router", messages= {"role": "user", "content": "Explain load balancing in one sentence."} , print resp.choices 0 .message.content Note the /v1 suffix on base url . LiteLLM happens to also expose some routes at the root, but the OpenAI SDK and most other OpenAI-compatible clients assumes the standard /v1 prefix and will build request paths accordingly. Pointing at root works by accident on LiteLLM specifically; pointing at /v1 works everywhere, so just do that. Run this a handful of times and you'll notice the model actually answering isn't always the same provider check the model field in the JSON response to confirm, LiteLLM rewrites it to the real underlying model it hit . Step 6: Check cost per request Run the curl command with -i as above and look at the response headers. LiteLLM adds x-litellm-response-cost , the dollar cost of that specific call, computed from the provider's published token pricing: curl -i http://localhost:4000/v1/chat/completions \ -H "Authorization: Bearer sk-gateway-1234" \ -H "Content-Type: application/json" \ -d '{"model": "smart-router", "messages": {"role": "user", "content": "hi"} }' \ | grep -i x-litellm For latency, time curl ... is enough to eyeball round-trip time locally. For real dashboards across many requests, you want Prometheus, covered in Next Steps. Verify it works - The proxy starts without errors and logs all three deployments under smart-router . - A curl request returns a normal OpenAI-shaped chat completion choices 0 .message.content . - The x-litellm-response-cost header is present and non-zero. - Temporarily set one provider's env var to an invalid value e.g. export OPENAI API KEY=invalid , restart the proxy, and send several requests. Some should still succeed, served by Claude or Gemini, proving the failover path actually works instead of just existing on paper. Troubleshooting : the AuthenticationError: No models match model name or similar model field in your request body has to exactly match a model name in config.yaml . Typos here are the most common failure. 401 from the gateway itself : you forgot Bearer in the Authorization header, or the token doesn't match LITELLM MASTER KEY . Note the env var is only read at proxy startup, restart after changing it. 401/403 bubbling up from a specific provider : usually a stale or mistyped key. Double-check the exact env var name LiteLLM expects per provider GEMINI API KEY , not GOOGLE API KEY , for the gemini/ prefix and confirm you exported it in the same shell you launched the proxy from. All requests failing after one provider goes down : check cooldown time and allowed fails , if they're too aggressive the whole group can get benched. Also confirm you actually have more than one deployment under the same model name , load balancing across a group of one does nothing. Next steps Latency-based routing : switch routing strategy to latency-based-routing once you're ready, but it needs a Redis instance to store rolling per-deployment latency across requests and processes a single in-memory process can't do this reliably at any real traffic level . Add redis host , redis port , and redis password under router settings and point them at a Redis container or managed instance. Persistent cost/spend tracking and the built-in UI : point LiteLLM at a Postgres database via DATABASE URL to unlock the /ui admin dashboard, virtual API keys, and per-team budgets, instead of reading cost off response headers one call at a time. Prometheus metrics : add prometheus to litellm settings.success callback to get a /metrics endpoint with latency histograms and request counts per model, then wire it into Grafana. Production deployment : run this behind a real reverse proxy with TLS, and read LiteLLM's docs on rate limiting and budget alerts before you put real traffic behind it. Rachel Goldstein https://sourcefeed.dev/u/rachel goldstein · Dev Tools Editor Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop. Discussion 0 No comments yet Be the first to weigh in.