# One model gateway across clouds: LiteLLM on Cloud Run and Azure with zero API keys

> Source: <https://dev.to/gde/one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api-keys-3ma7>
> Published: 2026-11-15 22:23:05+00:00

*As of November 2026. All identifiers in this post are examples.*

In [part 1](https://dev.to/migas_moreno/google-agent-platform-across-clouds-with-spiffe-zero-secrets-between-google-cloud-and-microsoft-1hjg) two AI agents, one on Google Cloud and one on Azure, learned to call each other without any stored secret.

Agents also call models. This post covers that half: a model gateway that runs in both clouds, serves Gemini and Azure OpenAI from either side, and holds no API key for any of it.

During a demo we switched off the gateway in one cloud. Nine seconds later the agent answered through the other one.

An agent can call a model directly. With two agents that is fine. With fifty it gets uncomfortable:

A model gateway fixes the three at once. Agents talk to one endpoint, and the rules and the usage records live there. We used [LiteLLM](https://docs.litellm.ai/) because it speaks to both Vertex AI and Azure OpenAI behind one OpenAI-compatible API.

The catch is that a gateway usually introduces two new kinds of secret: the keys it holds for the model providers, and the keys it hands out to its callers. We wanted neither.

There is one LiteLLM in each cloud, with the same configuration:

Each gateway is connected to both model families. The one on Azure can call Gemini, and the one on Google Cloud can call Azure OpenAI. An agent uses the gateway of its own cloud and gets the same catalogue either way.

The model list is the same file on both sides:

```
model_list:
  - model_name: gpt-4.1-mini
    litellm_params:
      model: azure/gpt-4.1-mini
      api_base: https://<your-resource>.openai.azure.com/
  - model_name: gemini-3.1-flash-lite
    litellm_params:
      model: vertex_ai/gemini-3.1-flash-lite
      vertex_project: <your-project>
      vertex_location: global
```

There is no `api_key` line anywhere in it.

LiteLLM normally gives each caller a virtual key. That is one more secret per agent, which is exactly what part 1 removed.

Each agent already has an identity in Microsoft Entra Agent ID and can get a token for any audience it has a role on. So the gateway is one more audience, `a2a-llm-gateway`, with one role, `LLM.Invoke`. The agent sends its own Entra token as the bearer.

On the LiteLLM side, a custom auth hook replaces the key lookup. Its core is short:

``` php
def _verify(token: str) -> dict:
    key = _JWKS.get_signing_key_from_jwt(token).key
    return jwt.decode(token, key, algorithms=["RS256"],
                      audience=AUDIENCES, issuer=ISSUER,
                      options={"require": ["exp", "iss", "aud"]})

async def user_api_key_auth(request, api_key):
    claims = await asyncio.to_thread(_verify, api_key)
    if "LLM.Invoke" not in (claims.get("roles") or []):
        raise Exception("forbidden: missing app role LLM.Invoke")
    team = await get_team_object(team_id=claims["azp"], ...)   # the agent's ID
    model = await _requested_model(request)
    if model and team.models and model not in team.models:
        raise Exception(f"forbidden: model '{model}' not allowed for this agent")
    return UserAPIKeyAuth(team_id=claims["azp"], team_models=team.models, ...)
```

It is wired in with one line of configuration:

```
general_settings:
  custom_auth: custom_auth.user_api_key_auth
```

The caller ID in the token is the agent's identity, and LiteLLM's "team" is keyed by that same ID. So usage is recorded per agent without anyone inventing a second naming scheme.

The allow-list comes from the same inventory file that part 1 used for onboarding:

```
- name: gcp-agent
  llm:
    gateway: gcp
    models: [gemini-3.1-flash-lite]
```

The onboarding pipeline assigns the `LLM.Invoke` role in Entra and creates the team in LiteLLM with those models. An agent that asks for a model outside its list is refused before any provider is called.

This is the part that makes the design hold together. Every link from the gateway outwards uses the identity of the workload.

**From Google Cloud to Gemini.** The Cloud Run service account has the Vertex AI role. Nothing to configure.

**From Google Cloud to Azure OpenAI.** LiteLLM asks Google for an OIDC token and presents it to Entra, where a federated credential on an app registration accepts it. In the model entry it is one line:

```
      azure_ad_token: oidc/google/api://AzureADTokenExchange
```

**From Azure to Azure OpenAI.** The Container App's managed identity.

**From Azure to Gemini.** This is the reverse federation. The managed identity's token is exchanged at a Google workload identity pool, which lets it act as the same service account the Cloud Run side uses. Google's client libraries do the exchange when they find this credential file:

```
{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/<NUMBER>/locations/global/workloadIdentityPools/azure-workloads/providers/<PROVIDER>",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": { "file": "/var/run/fed/azure-token" },
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/litellm-runtime@<PROJECT>.iam.gserviceaccount.com:generateAccessToken"
}
```

A small sidecar keeps the Azure token in that file fresh. The pool accepts only that one managed identity as a subject.

**To the database.** Both gateways reach Cloud SQL through the Cloud SQL Auth Proxy, as a sidecar, with IAM database authentication:

```
cloud-sql-proxy --auto-iam-authn --port=5432 <project>:<region>:<instance>
```

The database user is the service account. There is no password in the connection string.

Once every agent in a cloud depends on that cloud's gateway, the gateway is a single point of failure for all of them. So one thing has to be guaranteed: **an agent keeps getting answers when the gateway of its own cloud is unavailable.**

Two properties of this design make that possible:

There is also a rule worth stating: **a refusal is not an outage.** A 401, a 403 or a 429 means the gateway is working and said no. Moving to the other cloud must never be a way around that answer.

In our test, with the Azure gateway switched off, the Azure agent's next answer came through the Google Cloud gateway nine seconds later.

`DISABLE_SCHEMA_UPDATE=true`.` turn_off_message_logging: true` the database keeps who called what and how much, and never the prompts.
Between an agent and a model there is no API key to rotate.
