{"slug": "three-clouds-three-native-agents", "title": "Three Clouds, Three Native Agents", "summary": "A developer built a multi-cloud AI agent mesh using Google Cloud Run, AWS, and Azure, coordinated via the A2A protocol, and demonstrated that zero long-lived credentials are needed when the coordinator runs on Cloud Run. The project, available on GitHub, shows that federation via OIDC tokens eliminates static keys, though it limits local testing and makes one leg intra-cloud.", "body_md": "Three AI agents, each built with a different vendor's framework, each running on\n\nthat vendor's own hosting, all answering the same question at the same time:\n\nOne coordinator calls all three over **A2A v1.0** and takes the median of their\n\nanswers. And there is **no long-lived credential stored anywhere in the running\nsystem** — every call is authenticated with a token minted at the moment it is\n\nEverything is here:\n\n[github.com/xbill9/multicloud-adk-a2a-currency](https://github.com/xbill9/multicloud-adk-a2a-currency).\n\nYou can run the whole mesh on a laptop in about a minute; instructions are below.\n\nThe surprise wasn't the protocol. A2A worked. The surprise was that almost every\n\ndecision that mattered was made *before* a single A2A call happened.\n\nYou have an agent on one cloud. Someone asks you to have it call an agent on\n\nanother.\n\nThe reflex is to create a service account key, drop it in a secret manager, and\n\nmove on. That works. It also means you now own a credential forever — rotating\n\nit, scoping it, auditing it, and eventually explaining to somebody why\n\nproduction contains a static key.\n\nThere is another way, and the interesting part is that it isn't harder. It is\n\njust decided earlier.\n\nHere is the asymmetry the whole design falls out of.\n\n**Every agent you want to call can consume an external token.** AWS IAM has OIDC\n\nidentity providers. Entra has Federated Identity Credentials. AgentCore accepts a\n\n`CUSTOM_JWT`\n\n. All three will trust a token minted somewhere else, provided you\n\nset the trust up correctly.\n\n**But only some runtimes can mint one.** A runtime that can produce a workload\n\nOIDC token — for an audience *you* choose — can federate outward to any of them.\n\nA runtime that cannot is back to storing a credential.\n\nSo \"where does my coordinator run?\" is really \"how many secrets will this system\n\nhave?\"\n\n| Coordinator host | Legs it makes | Long-lived secrets |\n|---|---|---|\nCloud Run |\nGCP→AWS, GCP→Azure, GCP→GCP | potentially zero\n|\n| AgentCore | AWS→Azure, AWS→GCP | at least one |\n| Foundry | Azure→AWS, Azure→GCP | one or two, both unproven |\n\nCloud Run wins here because its metadata server hands you an ID token for any\n\naudience you name, which is exactly what the other two clouds' trust policies\n\nwant to see. Whether AgentCore can do the same is unconfirmed — I did not test\n\nit. So \"zero secrets\" is a property of *this* topology, not a law about\n\ncross-cloud agents.\n\nTwo things that choice costs you, worth saying out loud:\n\n**One leg stops being cross-cloud.** The coordinator runs on Cloud Run, so the\n\nGCP leg is Google calling Google. Two vendor boundaries get crossed, not three.\n\nThat belongs in the results, not in a footnote.\n\n**You cannot run it locally.** A user credential cannot mint an\n\narbitrary-audience ID token at all — `gcloud auth print-identity-token`\n\nrefuses outright, telling you it requires a service account.\n\n--audiences=...\n\nThere is no laptop version of this path. Once you choose federation, the only\n\nplace the system works is the place it is deployed.\n\nThe legs do not look alike:\n\n`roles/run.invoker`\n\n.`AssumeRoleWithWebIdentity`\n\n,\nget temporary credentials back, sign the request with SigV4.Two bearer tokens and a request signature. Different shapes entirely.\n\nThe move that made the rest tractable was putting all three behind one interface:\n\n`httpx.Auth`\n\n. To httpx, a bearer header and a signature over the request body are\n\nthe same kind of object. All three vendor SDKs accept an `httpx.AsyncClient`\n\n. So\n\nthe credential attaches once, and everything through that client carries it.\n\n```\nauth = credentials_for(peer, endpoint)   # an httpx.Auth, or None\nclient = load_client(stack, endpoint, auth=auth)\n```\n\nBuild that seam **before** your second cloud, not after your third. Get one leg\n\nworking with inline code and promise to generalise later, and you end up with\n\nthree error-handling styles and three places a token gets cached.\n\nWorth noticing:an agent's card lives at`/.well-known/agent-card.json`\n\n,\n\nand it sits behind the same authorization as the agent itself. Attach your\n\ncredential to therequestinstead of theclientand discovery 403s while\n\nthe actual call would have worked. You get a protocol error pointing nowhere\n\nnear auth. Attaching to the client makes that impossible by construction.\n\nNone of these are typos. Each is something you can get wrong while being careful.\n\n**Audience is not authorization.** The *caller* picks the audience. So a trust\n\npolicy checking only audience proves that *somebody* in that IdP minted a token —\n\nnot that *your* identity did. Pin the subject too, and pin it to the immutable\n\nnumeric ID rather than the email, because emails can be released and re-bound to\n\nsomeone else.\n\n**AWS and Azure invert the same step.** AWS federates with `accounts.google.com`\n\nnatively — create an explicit IAM OIDC provider for it and you *break* it with\n\n`InvalidIdentityToken`\n\n. Entra requires you to create the credential explicitly.\n\nSame conceptual task, opposite prerequisites, and neither error tells you which\n\nrule you are on.\n\n**The IAM condition keys do not hold what their names say.**\n\n`accounts.google.com:oaud`\n\nis the token's `aud`\n\n. `accounts.google.com:aud`\n\nis its\n\n`azp`\n\n, which is a number. Put an audience string in `:aud`\n\nand you have written a\n\ncondition that can never match. The denial will not mention it.\n\n**Ask for the whole token.** The GCP metadata mint takes `format=full`\n\n. Without\n\nit, Google trims claims — including `email`\n\n— and any trust condition reading\n\nthat claim silently stops matching.\n\n**Two error codes are worth more than a day of logging.** From STS,\n\n`InvalidIdentityToken`\n\nmeans the token did not validate at all, which is a\n\nprovider-setup problem. `AccessDenied`\n\nmeans it validated fine and your\n\nconditions did not match, which is a policy problem. Different afternoons.\n\nWhich leads to the one habit I would carry to any project like this: **log the\nraw provider response at every auth boundary.** In an agent system an error comes\n\n`AccessDenied: condition accounts.google.com:sub did not match`\n\ninto \"there wasStart local. Three agents on loopback, no cloud account, about a minute:\n\n```\ngit clone https://github.com/xbill9/multicloud-adk-a2a-currency\ncd multicloud-adk-a2a-currency\n\nuv pip install --system \"a2a-sdk[http-server]\" google-adk \\\n  agent-framework-a2a agent-framework-core \\\n  pydantic httpx uvicorn pytest pytest-asyncio\nuv pip install --system -e .\n```\n\nBring up the three agents and ask them a question:\n\n```\n./infra/run_mesh.sh start          # :10001 :10002 :10003\npython3 -m coordinator.cli 100 USD EUR JPY\n```\n\nThree vendors' agent stacks answering together:\n\n```\nparticipants: gcp, aws, azure\n\n100 USD = 92 EUR @ 0.92 [3/3 clouds, agreed]\n    gcp                  92 (164ms)\n    aws                  92 (25ms)\n    azure                92 (12ms)\n```\n\nThe demo is the more interesting run, because it shows what happens when a\n\nparticipant is *wrong*:\n\n```\n./infra/demo.sh\n```\n\nFour acts: three clouds answering, the 3×3 interop matrix, a cloud going\n\noffline, and a cloud lying. The last two are the point — anything can show three\n\ngreen ticks.\n\nDeploying for real is one script per cloud, then one command to wire them\n\ntogether:\n\n```\n./infra/deploy_aws.sh   deploy     # AgentCore Runtime + federated role\n./infra/deploy_azure.sh deploy     # Container App\n./infra/deploy_azure.sh fic        # Entra app registration + federated credential\n./infra/deploy_azure.sh auth       # make the ingress actually demand it\n\n./infra/deploy_gcp.sh deploy       # ADK service + coordinator job\n./infra/deploy_gcp.sh wire         # fold the AWS and Azure legs in\n./infra/deploy_gcp.sh run          # three-cloud consensus, from the cloud\n./infra/deploy_gcp.sh verify       # the negative controls\n```\n\nRunIt is the part that decides whether any of the auth`verify`\n\ntwice.\n\nclaims mean anything, for a reason covered below.\n\n**Put deployment in the repo as verbs, not in a runbook.** `deploy`\n\n, `wire`\n\n,\n\n`verify`\n\n. Each cloud's identifiers live in exactly one place — the script that\n\ncreated them — and the other scripts read them back rather than keeping copies.\n\nI can tell you precisely what that buys, because I tore the entire mesh down and\n\nrebuilt it from nothing to check.\n\nThe AWS runtime came back with a **different ARN**, and its invocation URL\n\ncontains that ARN. The Entra app registration came back with a **different client\nID**. The Container App came back on a\n\n`wire`\n\nread all three back out and the mesh returned:\n\n```\n100 USD = 92 EUR @ 0.92 [3/3 clouds, agreed]\n```\n\nAny copy of any of those identifiers stored anywhere else would have been stale\n\nthe moment it was written down.\n\nThen the whole verification pass ran again against infrastructure that had not\n\nexisted an hour earlier: three consensus runs at `3/3 clouds, agreed`\n\n, and all\n\neight auth probes — each leg answering with its credential, each leg denied\n\nwithout it, an unauthenticated request rejected, and a right-identity\n\nwrong-audience request rejected. Every number in this article comes from that\n\nrebuilt mesh.\n\nThat teardown also found two bugs that no amount of redeploying would have,\n\nbecause they live on code paths you can only reach from nothing:\n\n`None`\n\n. Under `set -e`\n\n, a `FlagMustBeSetForRestore`\n\n— an error that never mentions deletion. `destroy`\n\nfollowed by `deploy`\n\ncould not rebuild the Foundry account.\n\nIf you take one operational thing from this article:rebuild from nothing\n\nat least once before you tell anyone it is reproducible.\n\n**Scale to zero, and label what it costs.** Everything here idles at zero\n\nreplicas. Paying for idle capacity on three clouds to make a latency table look\n\ntidier is paying to mislead. But it means the first call into a leg pays a cold\n\nstart — a cold Azure leg measured **27.8 seconds** against **0.5 seconds** warm.\n\nMix those two regimes in one table and every conclusion drawn from it is wrong.\n\nFour structures did most of the work.\n\n| Structure | What it buys |\n|---|---|\nOne credential seam (`httpx.Auth` ) |\ncallers never know which of three mechanisms they are using |\nOne participant interface (`convert()` ) |\na cloud is an implementation, not a branch |\n| An instrument, not a demo | every failure typed by layer, not just red |\n| Controls scoped to one leg | a degrading system cannot hide a denial from you |\n\nThat last one is the one I would most want you to copy, because getting it wrong\n\nis invisible.\n\nThe mesh takes a median across three clouds and degrades on purpose. Lose a\n\ncloud, the other two still reach quorum, and the run exits **0**. Now try testing\n\nyour auth by removing one leg's credential from a three-cloud run. It still exits\n\nSo every leg gets probed alone. Eight probes: each leg answering with its\n\ncredential, each leg denied without it, an unauthenticated request rejected, and\n\na right-identity-wrong-audience request rejected. Only then does an exit code\n\nmean anything.\n\nThe general form: **any system with graceful degradation needs its controls\nscoped to a single component, or the degradation hides exactly the failure you\nare testing for.**\n\nWarm runs of the three-cloud consensus, after the rebuild:\n\n| GCP (in-cloud) | AWS | Azure | elapsed | |\n|---|---|---|---|---|\n| range | 836–948ms | 1027–1109ms | 468–512ms | 1711–1854ms |\n\nElapsed lands roughly a second above the *slowest single leg*, and far below the\n\nsum of all three. The legs are issued concurrently, so the sum was never the\n\nright model — but neither is the slowest leg on its own. That extra second is the\n\ncoordinator's own fixed cost: container start, three agent-card fetches, three\n\ncredential mints.\n\nWorth noticing:an earlier version of this claim quoted the slowest leg\n\nalone and waswrong by 85%on the fastest run. That error only became\n\nvisible once there was more than one sample.\n\nThe federation itself is cheap. Token mints and exchanges are a small slice of\n\nthat fixed second. If the mesh feels slow, it is a cold start or a model — not\n\nthe identity work.\n\nOne deployment, one account, one region pair, one person, over a few days. These\n\nare existence proofs: a thing worked, in a configuration. They are not\n\nmeasurements of a population.\n\nIt is keyless in operation, not in bootstrap. Creating trust policies, app\n\nregistrations and federated credentials used ordinary operator credentials, as\n\nprovisioning always does.\n\nAnd that claim needed checking, which is the honest part. The three A2A legs were\n\nalways keyless — but the Azure app pulled its container image using the\n\nregistry's admin password, stored as a secret in its own configuration. Not on\n\nany agent-to-agent path, and still enough to make \"no stored secrets\" false as\n\nwritten. Container Apps supports pulling by managed identity, so the fix was a\n\nrole grant and deleting the secret. An audit of all three deployments now shows\n\nno stored credential in any of them.\n\nThe dull general point: **image pull is part of your deployed system.** A claim\n\nabout secrets has to cover all of it, not just the interesting part.\n\nToken expiry and refresh are implemented and tested against a frozen clock, but\n\nno token has ever expired in production — every run is a job that lives a few\n\nseconds.\n\nDecide where the coordinator runs before anything else; it sets the secret count\n\nfor the entire system. Build the credential seam before the second cloud. Attach\n\nauth to the client, not the request, so discovery is covered. Log the provider's\n\nown words at every boundary, because you will spend more time reading auth\n\nfailures than writing auth code. Scope your controls to one component, because a\n\nsystem built to survive failure will happily hide one from you.\n\nAnd rebuild it from nothing once, before you claim it is reproducible.\n\n**Repo:**\n\n[github.com/xbill9/multicloud-adk-a2a-currency](https://github.com/xbill9/multicloud-adk-a2a-currency)\n\n— the three agents, the coordinator, the interop matrix, the deploy scripts, and\n\nthe findings write-ups in `docs/`\n\n.", "url": "https://wpnews.pro/news/three-clouds-three-native-agents", "canonical_source": "https://dev.to/aws-builders/three-clouds-three-native-agents-5hda", "published_at": "2026-08-10 15:04:44+00:00", "updated_at": "2026-08-10 15:17:38.136180+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Google Cloud Run", "AWS", "Azure", "A2A", "Cloud Run", "AgentCore", "Foundry", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/three-clouds-three-native-agents", "markdown": "https://wpnews.pro/news/three-clouds-three-native-agents.md", "text": "https://wpnews.pro/news/three-clouds-three-native-agents.txt", "jsonld": "https://wpnews.pro/news/three-clouds-three-native-agents.jsonld"}}