Stand up a local LiteLLM proxy that load-balances and fails over across OpenAI, Anthropic, and Gemini behind one OpenAI-compatible endpoint.
What you'll build #
A local LiteLLM proxy that exposes a single OpenAI-compatible /v1/chat/completions
endpoint. Behind that one endpoint, it load-balances requests across OpenAI, Anthropic, and Gemini, automatically retries on a different provider if one fails, and reports per-request cost. Your app code never has to know which provider actually served the request.
One naming note up front: this tutorial's title uses "GPT-5.6" as shorthand for "whichever GPT model is current when you read this." No such model has shipped. The working config below uses gpt-4o
, a current OpenAI production model at the time of writing, so every command here actually runs. LiteLLM doesn't validate model strings itself, it just passes whatever you put in model:
straight to the provider, so swapping in a newer OpenAI model later means changing one line and nothing else.
Prerequisites #
- Python 3.9 or newer (
python3 --version
) - API keys for the providers you want behind the gateway: an OpenAI key, an Anthropic key, and a Google AI Studio (Gemini) key
- macOS, Linux, or WSL2 on Windows. No compiled dependencies here, so Apple Silicon vs Intel doesn't matter
- Basic comfort with YAML and curl
Step 1: Install LiteLLM #
Use a virtual environment so this doesn't collide with other Python projects.
python3 -m venv venv
source venv/bin/activate
pip install 'litellm[proxy]'
Confirm it installed:
pip show litellm
You should see a Version:
line. That's enough to confirm the install worked, don't rely on a --version
flag on the litellm
CLI itself, since not every build supports it consistently.
If you'd rather skip the local Python environment entirely, the official Docker image works the same way:
docker pull ghcr.io/berriai/litellm:main-latest
Step 2: Set your provider credentials #
LiteLLM reads provider keys from environment variables. Export them in the shell where you'll run the proxy:
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="AIza..."
export LITELLM_MASTER_KEY="sk-gateway-1234"
LITELLM_MASTER_KEY
isn't a provider key, it's the credential clients will use to authenticate to your gateway. Pick a random string, don't reuse a provider key here.
Don't put any of these in a committed file. If you want persistence across shells, put the exports in a .env
file that's in your .gitignore
, or use your OS keychain.
Step 3: Write the gateway config #
Create config.yaml
:
model_list:
- model_name: smart-router
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: smart-router
litellm_params:
model: anthropic/claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: smart-router
litellm_params:
model: gemini/gemini-2.0-flash
api_key: os.environ/GEMINI_API_KEY
router_settings:
routing_strategy: simple-shuffle
num_retries: 2
timeout: 30
allowed_fails: 2
cooldown_time: 30
litellm_settings:
drop_params: true
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
A few things worth understanding here, not just copying:
- Giving all three deployments the same
model_name
(smart-router
) is what makes them one logical group. Clients callsmart-router
, LiteLLM picks a deployment from the group and load-balances across them. routing_strategy: simple-shuffle
is the default and requires no external infrastructure. It picks randomly (weighted if you setweight
per deployment) among healthy deployments. Don't reach forlatency-based-routing
yet, it needs a Redis backend to store rolling latency stats across requests, more on that in Next Steps.num_retries
andallowed_fails
are your failover knobs. If the OpenAI deployment throws an error, LiteLLM retries against Claude or Gemini in the same group instead of just failing the request.cooldown_time
temporarily benches a deployment that's been erroring, so you're not hammering a provider that's down.drop_params: true
strips provider-unsupported request params instead of erroring, useful since not every provider accepts every OpenAI parameter (likelogprobs
on some Gemini models).
If you'd rather use a smaller, cheaper OpenAI model for this exercise, swap openai/gpt-4o
for openai/gpt-4o-mini
, both are current and work identically here.
Step 4: Start the proxy #
litellm --config config.yaml --port 4000
You should see log lines confirming each deployment loaded, plus a line saying the proxy is running on http://0.0.0.0:4000
. Leave this running in its own terminal tab.
Docker equivalent, mounting your config in:
docker run -p 4000:4000 \
-e OPENAI_API_KEY -e ANTHROPIC_API_KEY -e GEMINI_API_KEY -e LITELLM_MASTER_KEY \
-v $(pwd)/config.yaml:/app/config.yaml \
ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --port 4000
Step 5: Call it like any OpenAI endpoint #
Raw curl:
curl -i http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-gateway-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "smart-router",
"messages": [{"role": "user", "content": "Say hello in exactly 5 words."}]
}'
Or from Python, using the official openai
SDK unchanged except for base_url
:
from openai import OpenAI
client = OpenAI(
api_key="sk-gateway-1234",
base_url="http://localhost:4000/v1",
)
resp = client.chat.completions.create(
model="smart-router",
messages=[{"role": "user", "content": "Explain load balancing in one sentence."}],
)
print(resp.choices[0].message.content)
Note the /v1
suffix on base_url
. LiteLLM happens to also expose some routes at the root, but the OpenAI SDK (and most other OpenAI-compatible clients) assumes the standard /v1
prefix and will build request paths accordingly. Pointing at root works by accident on LiteLLM specifically; pointing at /v1
works everywhere, so just do that.
Run this a handful of times and you'll notice the model actually answering isn't always the same provider (check the model
field in the JSON response to confirm, LiteLLM rewrites it to the real underlying model it hit).
Step 6: Check cost per request #
Run the curl command with -i
(as above) and look at the response headers. LiteLLM adds x-litellm-response-cost
, the dollar cost of that specific call, computed from the provider's published token pricing:
curl -i http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer sk-gateway-1234" \
-H "Content-Type: application/json" \
-d '{"model": "smart-router", "messages": [{"role": "user", "content": "hi"}]}' \
| grep -i x-litellm
For latency, time curl ...
is enough to eyeball round-trip time locally. For real dashboards across many requests, you want Prometheus, covered in Next Steps.
Verify it works #
- The proxy starts without errors and logs all three deployments under
smart-router
. - A curl request returns a normal OpenAI-shaped chat completion (
choices[0].message.content
). - The
x-litellm-response-cost
header is present and non-zero. - Temporarily set one provider's env var to an invalid value (e.g.
export OPENAI_API_KEY=invalid
), restart the proxy, and send several requests. Some should still succeed, served by Claude or Gemini, proving the failover path actually works instead of just existing on paper.
Troubleshooting #
: theAuthenticationError: No models match model_name
or similarmodel
field in your request body has to exactly match amodel_name
inconfig.yaml
. Typos here are the most common failure.401 from the gateway itself: you forgotBearer
in theAuthorization
header, or the token doesn't matchLITELLM_MASTER_KEY
. Note the env var is only read at proxy startup, restart after changing it.401/403 bubbling up from a specific provider: usually a stale or mistyped key. Double-check the exact env var name LiteLLM expects per provider (GEMINI_API_KEY
, notGOOGLE_API_KEY
, for thegemini/
prefix) and confirm you exported it in the same shell you launched the proxy from.All requests failing after one provider goes down: checkcooldown_time
andallowed_fails
, if they're too aggressive the whole group can get benched. Also confirm you actually have more than one deployment under the samemodel_name
, load balancing across a group of one does nothing.
Next steps #
Latency-based routing: switchrouting_strategy
tolatency-based-routing
once you're ready, but it needs a Redis instance to store rolling per-deployment latency across requests and processes (a single in-memory process can't do this reliably at any real traffic level). Addredis_host
,redis_port
, andredis_password
underrouter_settings
and point them at a Redis container or managed instance.Persistent cost/spend tracking and the built-in UI: point LiteLLM at a Postgres database viaDATABASE_URL
to unlock the/ui
admin dashboard, virtual API keys, and per-team budgets, instead of reading cost off response headers one call at a time.Prometheus metrics: addprometheus
tolitellm_settings.success_callback
to get a/metrics
endpoint with latency histograms and request counts per model, then wire it into Grafana.Production deployment: run this behind a real reverse proxy with TLS, and read LiteLLM's docs on rate limiting and budget alerts before you put real traffic behind it.
Rachel Goldstein· Dev Tools Editor
Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.
Discussion 0 #
No comments yet
Be the first to weigh in.