{"slug": "one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api", "title": "One model gateway across clouds: LiteLLM on Cloud Run and Azure with zero API keys", "summary": "A developer built a LiteLLM-based model gateway deployed on both Google Cloud Run and Azure that serves Gemini and Azure OpenAI through a single OpenAI-compatible endpoint without storing any provider API keys. Authentication relies on Microsoft Entra Agent ID tokens verified via a custom LiteLLM auth hook against JWKS, with per-agent model allow-lists enforced before any provider call. In a demo, shutting down the gateway in one cloud caused the agent to answer through the other within nine seconds.", "body_md": "*As of November 2026. All identifiers in this post are examples.*\n\nIn [part 1](https://dev.to/migas_moreno/google-agent-platform-across-clouds-with-spiffe-zero-secrets-between-google-cloud-and-microsoft-1hjg) two AI agents, one on Google Cloud and one on Azure, learned to call each other without any stored secret.\n\nAgents also call models. This post covers that half: a model gateway that runs in both clouds, serves Gemini and Azure OpenAI from either side, and holds no API key for any of it.\n\nDuring a demo we switched off the gateway in one cloud. Nine seconds later the agent answered through the other one.\n\nAn agent can call a model directly. With two agents that is fine. With fifty it gets uncomfortable:\n\nA model gateway fixes the three at once. Agents talk to one endpoint, and the rules and the usage records live there. We used [LiteLLM](https://docs.litellm.ai/) because it speaks to both Vertex AI and Azure OpenAI behind one OpenAI-compatible API.\n\nThe catch is that a gateway usually introduces two new kinds of secret: the keys it holds for the model providers, and the keys it hands out to its callers. We wanted neither.\n\nThere is one LiteLLM in each cloud, with the same configuration:\n\nEach gateway is connected to both model families. The one on Azure can call Gemini, and the one on Google Cloud can call Azure OpenAI. An agent uses the gateway of its own cloud and gets the same catalogue either way.\n\nThe model list is the same file on both sides:\n\n```\nmodel_list:\n  - model_name: gpt-4.1-mini\n    litellm_params:\n      model: azure/gpt-4.1-mini\n      api_base: https://<your-resource>.openai.azure.com/\n  - model_name: gemini-3.1-flash-lite\n    litellm_params:\n      model: vertex_ai/gemini-3.1-flash-lite\n      vertex_project: <your-project>\n      vertex_location: global\n```\n\nThere is no `api_key` line anywhere in it.\n\nLiteLLM normally gives each caller a virtual key. That is one more secret per agent, which is exactly what part 1 removed.\n\nEach agent already has an identity in Microsoft Entra Agent ID and can get a token for any audience it has a role on. So the gateway is one more audience, `a2a-llm-gateway`, with one role, `LLM.Invoke`. The agent sends its own Entra token as the bearer.\n\nOn the LiteLLM side, a custom auth hook replaces the key lookup. Its core is short:\n\n``` php\ndef _verify(token: str) -> dict:\n    key = _JWKS.get_signing_key_from_jwt(token).key\n    return jwt.decode(token, key, algorithms=[\"RS256\"],\n                      audience=AUDIENCES, issuer=ISSUER,\n                      options={\"require\": [\"exp\", \"iss\", \"aud\"]})\n\nasync def user_api_key_auth(request, api_key):\n    claims = await asyncio.to_thread(_verify, api_key)\n    if \"LLM.Invoke\" not in (claims.get(\"roles\") or []):\n        raise Exception(\"forbidden: missing app role LLM.Invoke\")\n    team = await get_team_object(team_id=claims[\"azp\"], ...)   # the agent's ID\n    model = await _requested_model(request)\n    if model and team.models and model not in team.models:\n        raise Exception(f\"forbidden: model '{model}' not allowed for this agent\")\n    return UserAPIKeyAuth(team_id=claims[\"azp\"], team_models=team.models, ...)\n```\n\nIt is wired in with one line of configuration:\n\n```\ngeneral_settings:\n  custom_auth: custom_auth.user_api_key_auth\n```\n\nThe caller ID in the token is the agent's identity, and LiteLLM's \"team\" is keyed by that same ID. So usage is recorded per agent without anyone inventing a second naming scheme.\n\nThe allow-list comes from the same inventory file that part 1 used for onboarding:\n\n```\n- name: gcp-agent\n  llm:\n    gateway: gcp\n    models: [gemini-3.1-flash-lite]\n```\n\nThe onboarding pipeline assigns the `LLM.Invoke` role in Entra and creates the team in LiteLLM with those models. An agent that asks for a model outside its list is refused before any provider is called.\n\nThis is the part that makes the design hold together. Every link from the gateway outwards uses the identity of the workload.\n\n**From Google Cloud to Gemini.** The Cloud Run service account has the Vertex AI role. Nothing to configure.\n\n**From Google Cloud to Azure OpenAI.** LiteLLM asks Google for an OIDC token and presents it to Entra, where a federated credential on an app registration accepts it. In the model entry it is one line:\n\n```\n      azure_ad_token: oidc/google/api://AzureADTokenExchange\n```\n\n**From Azure to Azure OpenAI.** The Container App's managed identity.\n\n**From Azure to Gemini.** This is the reverse federation. The managed identity's token is exchanged at a Google workload identity pool, which lets it act as the same service account the Cloud Run side uses. Google's client libraries do the exchange when they find this credential file:\n\n```\n{\n  \"type\": \"external_account\",\n  \"audience\": \"//iam.googleapis.com/projects/<NUMBER>/locations/global/workloadIdentityPools/azure-workloads/providers/<PROVIDER>\",\n  \"subject_token_type\": \"urn:ietf:params:oauth:token-type:jwt\",\n  \"token_url\": \"https://sts.googleapis.com/v1/token\",\n  \"credential_source\": { \"file\": \"/var/run/fed/azure-token\" },\n  \"service_account_impersonation_url\": \"https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/litellm-runtime@<PROJECT>.iam.gserviceaccount.com:generateAccessToken\"\n}\n```\n\nA small sidecar keeps the Azure token in that file fresh. The pool accepts only that one managed identity as a subject.\n\n**To the database.** Both gateways reach Cloud SQL through the Cloud SQL Auth Proxy, as a sidecar, with IAM database authentication:\n\n```\ncloud-sql-proxy --auto-iam-authn --port=5432 <project>:<region>:<instance>\n```\n\nThe database user is the service account. There is no password in the connection string.\n\nOnce every agent in a cloud depends on that cloud's gateway, the gateway is a single point of failure for all of them. So one thing has to be guaranteed: **an agent keeps getting answers when the gateway of its own cloud is unavailable.**\n\nTwo properties of this design make that possible:\n\nThere is also a rule worth stating: **a refusal is not an outage.** A 401, a 403 or a 429 means the gateway is working and said no. Moving to the other cloud must never be a way around that answer.\n\nIn our test, with the Azure gateway switched off, the Azure agent's next answer came through the Google Cloud gateway nine seconds later.\n\n`DISABLE_SCHEMA_UPDATE=true`.` turn_off_message_logging: true` the database keeps who called what and how much, and never the prompts.\nBetween an agent and a model there is no API key to rotate.", "url": "https://wpnews.pro/news/one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api", "canonical_source": "https://dev.to/gde/one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api-keys-3ma7", "published_at": "2026-11-15 22:23:05+00:00", "updated_at": "2026-11-15 22:25:05.463048+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["LiteLLM", "Google Cloud Run", "Microsoft Azure", "Azure OpenAI", "Gemini", "Vertex AI", "Microsoft Entra Agent ID"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api", "markdown": "https://wpnews.pro/news/one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api.md", "text": "https://wpnews.pro/news/one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api.txt", "jsonld": "https://wpnews.pro/news/one-model-gateway-across-clouds-litellm-on-cloud-run-and-azure-with-zero-api.jsonld"}}