cd /news/ai-agents/one-model-gateway-across-clouds-lite… · home › topics › ai-agents › article
[ARTICLE · art-170146] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

One model gateway across clouds: LiteLLM on Cloud Run and Azure with zero API keys

A developer built a LiteLLM-based model gateway deployed on both Google Cloud Run and Azure that serves Gemini and Azure OpenAI through a single OpenAI-compatible endpoint without storing any provider API keys. Authentication relies on Microsoft Entra Agent ID tokens verified via a custom LiteLLM auth hook against JWKS, with per-agent model allow-lists enforced before any provider call. In a demo, shutting down the gateway in one cloud caused the agent to answer through the other within nine seconds.

by read5 min views1 publishedNov 15, 2026

As of November 2026. All identifiers in this post are examples.

In part 1 two AI agents, one on Google Cloud and one on Azure, learned to call each other without any stored secret.

Agents also call models. This post covers that half: a model gateway that runs in both clouds, serves Gemini and Azure OpenAI from either side, and holds no API key for any of it.

During a demo we switched off the gateway in one cloud. Nine seconds later the agent answered through the other one.

An agent can call a model directly. With two agents that is fine. With fifty it gets uncomfortable:

A model gateway fixes the three at once. Agents talk to one endpoint, and the rules and the usage records live there. We used LiteLLM because it speaks to both Vertex AI and Azure OpenAI behind one OpenAI-compatible API.

The catch is that a gateway usually introduces two new kinds of secret: the keys it holds for the model providers, and the keys it hands out to its callers. We wanted neither.

There is one LiteLLM in each cloud, with the same configuration:

Each gateway is connected to both model families. The one on Azure can call Gemini, and the one on Google Cloud can call Azure OpenAI. An agent uses the gateway of its own cloud and gets the same catalogue either way.

The model list is the same file on both sides:

model_list:
  - model_name: gpt-4.1-mini
    litellm_params:
      model: azure/gpt-4.1-mini
      api_base: https://<your-resource>.openai.azure.com/
  - model_name: gemini-3.1-flash-lite
    litellm_params:
      model: vertex_ai/gemini-3.1-flash-lite
      vertex_project: <your-project>
      vertex_location: global

There is no api_key line anywhere in it.

LiteLLM normally gives each caller a virtual key. That is one more secret per agent, which is exactly what part 1 removed.

Each agent already has an identity in Microsoft Entra Agent ID and can get a token for any audience it has a role on. So the gateway is one more audience, a2a-llm-gateway, with one role, LLM.Invoke. The agent sends its own Entra token as the bearer.

On the LiteLLM side, a custom auth hook replaces the key lookup. Its core is short:

def _verify(token: str) -> dict:
    key = _JWKS.get_signing_key_from_jwt(token).key
    return jwt.decode(token, key, algorithms=["RS256"],
                      audience=AUDIENCES, issuer=ISSUER,
                      options={"require": ["exp", "iss", "aud"]})

async def user_api_key_auth(request, api_key):
    claims = await asyncio.to_thread(_verify, api_key)
    if "LLM.Invoke" not in (claims.get("roles") or []):
        raise Exception("forbidden: missing app role LLM.Invoke")
    team = await get_team_object(team_id=claims["azp"], ...)   # the agent's ID
    model = await _requested_model(request)
    if model and team.models and model not in team.models:
        raise Exception(f"forbidden: model '{model}' not allowed for this agent")
    return UserAPIKeyAuth(team_id=claims["azp"], team_models=team.models, ...)

It is wired in with one line of configuration:

general_settings:
  custom_auth: custom_auth.user_api_key_auth

The caller ID in the token is the agent's identity, and LiteLLM's "team" is keyed by that same ID. So usage is recorded per agent without anyone inventing a second naming scheme.

The allow-list comes from the same inventory file that part 1 used for onboarding:

- name: gcp-agent
  llm:
    gateway: gcp
    models: [gemini-3.1-flash-lite]

The onboarding pipeline assigns the LLM.Invoke role in Entra and creates the team in LiteLLM with those models. An agent that asks for a model outside its list is refused before any provider is called.

This is the part that makes the design hold together. Every link from the gateway outwards uses the identity of the workload.

From Google Cloud to Gemini. The Cloud Run service account has the Vertex AI role. Nothing to configure.

From Google Cloud to Azure OpenAI. LiteLLM asks Google for an OIDC token and presents it to Entra, where a federated credential on an app registration accepts it. In the model entry it is one line:

      azure_ad_token: oidc/google/api://AzureADTokenExchange

From Azure to Azure OpenAI. The Container App's managed identity.

From Azure to Gemini. This is the reverse federation. The managed identity's token is exchanged at a Google workload identity pool, which lets it act as the same service account the Cloud Run side uses. Google's client libraries do the exchange when they find this credential file:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/<NUMBER>/locations/global/workloadIdentityPools/azure-workloads/providers/<PROVIDER>",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": { "file": "/var/run/fed/azure-token" },
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/litellm-runtime@<PROJECT>.iam.gserviceaccount.com:generateAccessToken"
}

A small sidecar keeps the Azure token in that file fresh. The pool accepts only that one managed identity as a subject.

To the database. Both gateways reach Cloud SQL through the Cloud SQL Auth Proxy, as a sidecar, with IAM database authentication:

cloud-sql-proxy --auto-iam-authn --port=5432 <project>:<region>:<instance>

The database user is the service account. There is no password in the connection string.

Once every agent in a cloud depends on that cloud's gateway, the gateway is a single point of failure for all of them. So one thing has to be guaranteed: an agent keeps getting answers when the gateway of its own cloud is unavailable.

Two properties of this design make that possible:

There is also a rule worth stating: a refusal is not an outage. A 401, a 403 or a 429 means the gateway is working and said no. Moving to the other cloud must never be a way around that answer.

In our test, with the Azure gateway switched off, the Azure agent's next answer came through the Google Cloud gateway nine seconds later.

DISABLE_SCHEMA_UPDATE=true. turn_off_message_logging: true the database keeps who called what and how much, and never the prompts. Between an agent and a model there is no API key to rotate.

── more in #ai-agents 4 stories · sorted by recency
── more on @litellm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/one-model-gateway-ac…] indexed:0 read:5min 2026-11-15 · —