Everything That Goes Wrong With That Setup (And How I Debugged It)

A developer debugging a Gemini on Vertex AI deployment encountered six distinct bugs, each initially appearing as a 401 Unauthorized error. The issues ranged from a PM2 configuration typo causing a 502 Bad Gateway to the Google Cloud client library incorrectly querying the AWS Instance Metadata Service, which returned a generic XHTML 401 page. The developer systematically ruled out each problem by isolating the root cause, such as verifying environment variables with `pm2 env 0` and testing IMDS directly.

In Part 1 https://dev.to/prathamchauhan/setting-up-gemini-on-vertex-ai-for-production-a-no-nonsense-walkthrough-22hp , I walked through setting up Gemini via Vertex AI, from project setup through a keyless EC2 deployment with Workload Identity Federation WIF . If a few of those terms went by fast, quick recap: a service account is a non-human identity your app logs in as, an IAM role is a labeled bundle of permissions, and WIF is the system that lets AWS vouch for your server's identity to Google without any password changing hands. This post is the part nobody puts in the official docs: every single thing that went wrong when I actually did this, in the order I hit it, and what each error really meant. Six distinct bugs, stacked on top of each other. Almost every one of them initially looked like a variation of the same "401 Unauthorized" error, and figuring out that they were six different problems, not one stubborn one, was most of the battle. Before I'd even gotten to testing auth, the site returned a 502 Bad Gateway. Easy mistake to make here: jumping straight to "something's wrong with my Vertex AI setup." A 502 just means the web server in front of your app commonly nginx sent a request to your app and got no answer back at all. It tells you nothing about why, only that your app isn't there to ask. pm2 list pm2 logs <app-name --lines 200 curl -v http://localhost:3000 If curl localhost:3000 doesn't get a response, the problem is the Node process itself, full stop, not anything cloud related. Always rule this out first. The actual cause of that 502: I'd saved my PM2 ecosystem file, which is just a config file that tells PM2 how to start and manage your app, as ecosystem.config.cjs correct, since module.exports is older CommonJS style syntax but ran: pm2 start ecosystem.config.js filename typo, PM2 never loaded it PM2 didn't throw an error. It just fell back to running the app a different way, which meant none of my environment variables, things like GOOGLE APPLICATION CREDENTIALS , project ID, location, ever reached the process. The app looked alive. It just had none of the config I assumed it had. pm2 delete <app-name pm2 start ecosystem.config.cjs pm2 env 0 confirm the vars you expect are actually there That last command became my go to sanity check for the rest of this whole debugging session, and honestly should be step one any time an app "isn't picking up" an environment variable. With the app finally up, I got a real, recognizable Google error: {"error": "Could not load the default credentials..."} Straightforward enough. But a bit later, a completely different looking error showed up: < DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" ... <title 401 - Unauthorized</title Here's the detail that mattered: this isn't a shape Google ever returns. Google's API errors are JSON. Their HTML error pages are branded with Google's actual logo and styling. This bare, unbranded XHTML 401 page is a generic default error document, the kind produced by a totally different piece of infrastructure. I burned real time suspecting a corporate proxy sitting between my server and the internet, then suspecting AWS S3, since the app also uploads images there. Both wrong. Direct curl tests against Google and S3 came back clean, with real certificates and their own genuine error formats. The actual source: http://169.254.169.254/latest/meta-data/iam/security-credentials/ . That address is the AWS EC2 Instance Metadata Service, usually shortened to IMDS. Every AWS EC2 server can quietly ask this internal, invisible to the outside world address "who am I, and what permissions do I currently have," without needing a password to ask. curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ → the exact same XHTML 401 page Worth explaining why a Google library was even talking to an AWS only address. That same 169.254.169.254 address is the standard link local metadata address on both AWS and Google Cloud. It's not a coincidence, it's part of a convention both cloud providers adopted. Google's auth library has a built in fallback that checks this exact address when it can't find credentials another way, because on a real Google Cloud server, that's genuinely where instance metadata lives. Run that same library on an EC2 instance instead, and it knocks on the same door, except now AWS is answering, with AWS's rules, not Google's. That collision is the single most confusing part of this entire setup, and the reason a "Google auth error" turned out to actually be an AWS one. Takeaway: when an error's shape doesn't match what the service "should" return, trust that mismatch. It's telling you to look elsewhere. Once I knew it was IMDS, the next question was why it was rejecting the request. My EC2 instance correctly enforced HttpTokens: required , which means it was running the newer, safer version of the metadata service called IMDSv2. In plain terms, IMDSv1 let absolutely anything on the machine ask the metadata address a question and get an answer back immediately. IMDSv2 adds one extra step: you first have to ask for a short lived "session token" with a separate request, and only then can you use that token to actually ask your question. It exists specifically to block a class of attack where a vulnerability in your app could otherwise be tricked into reading your server's cloud credentials. Disabling it is not the fix. google-auth-library does support IMDSv2, but only if your WIF credential config explicitly tells it to, via one field inside credential source , which is the part of the config file describing exactly how to fetch AWS's identity proof: "credential source": { "environment id": "aws1", "region url": "http://169.254.169.254/latest/meta-data/placement/availability-zone", "url": "http://169.254.169.254/latest/meta-data/iam/security-credentials", "regional cred verification url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15", "imdsv2 session token url": "http://169.254.169.254/latest/api/token" } My config, generated before I'd thought hard about IMDSv2, was missing that one line. Add it, and the library correctly fetches that short lived session token first before making any metadata requests, satisfying IMDSv2 without weakening it. One thing to watch for down the road: there's a known upstream bug where this session token gets cached with only a 300 second lifespan, which can cause intermittent failures on long running processes well after everything looks fixed. If the same error reappears hours or days later, that's the first thing to check, not a regression in your config. Auth chain working, AWS identity exchanged for a Google token, and a brand new error appeared: Permission 'aiplatform.endpoints.predict' denied... or it may not exist I tried granting IAM roles to gemini-prod-runner@my-project.iam.gserviceaccount.com , per Part 1's instructions, and got: INVALID ARGUMENT: Service account gemini-prod-runner@... does not exist. The account in my actual project had drifted to a slightly different name during earlier setup. Worse, my WIF config had no service account impersonation url field at all. Impersonation, in this context, just means "let identity A borrow identity B's permissions temporarily." Its absence meant my setup wasn't using that pattern in the first place. It was mapping the AWS identity straight to a Google identity of its own, with no service account in between. gcloud iam service-accounts list --project="$PROJECT ID" cat /path/to/credentials.json check for service account impersonation url Lesson: don't assume your WIF setup follows the impersonation pattern just because that's the common example in tutorials, including Part 1's. Read your actual credential config file before granting anything. It tells you exactly which pattern you're on. The final piece: figuring out what string to actually grant roles/aiplatform.user to, since no service account was involved. This came down to the provider's attribute mapping, which is just the rule set that tells Google how to translate the raw identity info AWS sends over into something Google understands: attributeMapping: attribute.aws role: "assertion.arn.contains 'assumed-role' ? ... : assertion.arn" google.subject: assertion.arn A quick translation of the jargon here. An ARN is AWS's way of uniquely naming any resource, including an identity, written as a long string like arn:aws:sts::123456789:assumed-role/my-role/instance-id . The "assertion" is the signed proof AWS hands over saying "this is genuinely who's asking." google.subject is Google's term for the exact identity string it sees. I could have bound the role to google.subject , the full AWS identity string including the specific EC2 instance ID baked into it. That would have worked, but only for that one instance, breaking the moment it got replaced or the app scaled to a second server. Binding to attribute.aws role instead, a cleaned up version of that string with the instance specific part stripped out, targets the IAM role generally. That means any current or future instance carrying that role authenticates correctly with no further changes: gcloud projects add-iam-policy-binding "$PROJECT ID" \ --member="principalSet://iam.googleapis.com/projects/PROJECT NUMBER/locations/global/workloadIdentityPools/aws-ec2-pool/attribute.aws role/arn:aws:sts::AWS ACCOUNT ID:assumed-role/ec2-gemini-runner" \ --role="roles/aiplatform.user" One more bit of jargon worth untangling here: principal versus principalSet . A principal is one single, specific identity, like one exact instance's session. A principalSet is a whole group of identities that share some attribute, in this case "anyone using this AWS role," no matter which instance they're running on. That's why principalSet was the right choice for something durable. That was the last piece. Gemini finally responded. If I were setting this up again, knowing what I know now: I'd get the app fully working with a plain service account JSON key first, and only swap to WIF as a deliberate, isolated second step, since debugging two unfamiliar systems at the same time, your app's Vertex AI integration and a federated trust chain between two clouds, is much harder than debugging them one at a time. I'd open the credential config file before assuming anything about impersonation, since it tells you definitively which auth pattern you're on. I'd compare error response shapes, not just status codes, since a 401 from Google, a 401 from AWS IMDS, and a 401 from anything in between all look different once you actually read them side by side. I'd always fully restart my process manager, meaning delete and start fresh rather than just restart, after any environment variable change, and verify with pm2 env . And I'd follow up every add-iam-policy-binding command with a get-iam-policy check, since a typo'd identity name fails loudly but a propagation delay or a wrong project fails silently. None of these six bugs were individually hard. Stacked on top of each other, each one disguised as a slightly different flavor of the same 401, they ate a full day. Hopefully Part 1 plus this post saves you most of that day.