How I Debugged a Broken Vertex AI Image Pipeline and Finally Made It Reliable

A developer detailed debugging a broken Vertex AI image pipeline, finding that most failures were credential, project, and policy issues rather than model problems. The fix involved correcting a hardcoded project ID, separating free and paid routes, and testing token minting before touching prompts. The developer emphasized that the pipeline was only considered fixed when it produced an actual image file.

Last week, I lost more time to AI infrastructure than to the actual creative work. The task sounded simple: generate images through Vertex AI for a content pipeline. What actually happened: invalid grant permission denied This is the part nobody tells you about AI workflows: most failures are not model failures. They’re credential, project, and policy failures. Here’s how I finally debugged the whole thing and got the image route working again. At first, the failures looked unrelated. I saw three different classes of errors: invalid grant: account not found 403 Permission denied 429 RESOURCE EXHAUSTED That usually means one of two things: In my case, it was the second one. The first useful move was brutally simple: find the one script that the team actually trusts. For us, that was: ~/clawd/ops/production/scripts/generate panels.py That became the source of truth. Not old snippets. Not half-working notebooks. Not memory. Once I checked the actual script, I immediately found one hidden problem: PROJECT = "old-project-id" The pipeline was still hardcoded to an old project. So even when I updated the credentials, the requests were still going to the wrong place. That alone explained a lot. We had two different routes mixed together: That sounds harmless, but it creates terrible debugging conditions. Because the failure modes are different: If you mix them, you start solving the wrong problem. For example, this looked like a model problem at first: 429 RESOURCE EXHAUSTED But it turned out to be just a burned free-tier key. Meanwhile the paid route was failing for a completely different reason. Lesson: treat free and paid as separate systems, even if they use the same model. Once I had the new Vertex JSON, I didn’t start by generating an image. I started by checking whether the credential could even mint a token. That kind of test saves time because it tells you whether the problem is: In Python, the logic is basically: python from google.oauth2 import service account from google.auth.transport.requests import Request creds = service account.Credentials.from service account file "vertex ai key.json", scopes= "https://www.googleapis.com/auth/cloud-platform" , creds.refresh Request print creds.token :40 If this step fails, don’t touch your prompts. Don’t touch the model. Don’t touch the rendering code. You don’t have an image problem yet. You have an auth problem. This one burned the most time. I created a new service account, everything looked correct, and then Google Cloud refused to create a JSON key. The error turned out to be caused by this policy: iam.disableServiceAccountKeyCreation That wasn’t obvious from the first screen. The UI showed one policy as “not enforced,” while a legacy constraint was still active somewhere above it. That kind of mismatch is why cloud debugging feels cursed. The practical fix was not to keep fighting the same project. The practical fix was to create a clean personal project without inherited org-policy baggage. That ended up being faster than trying to untangle admin policy state. The final working setup looked like this: Only after that did I consider the route fixed. Not when the key existed. Not when the policy screen looked green. Not when the script stopped crashing. Only when this produced an actual file: outputs/nanobanana vertex test.png That was the only result that mattered. When an AI image pipeline breaks, I now check things in this order: That order is much faster than randomly changing keys and re-running prompts. For us, the final fix was not “better prompting.” It was: That’s not glamorous. But it’s the difference between a pipeline you trust and a pipeline that only works when you’re lucky. A lot of AI tooling discourse is still obsessed with models. But once you work with these systems in production, the real bottleneck is often much more boring: identity, permissions, quotas, and project hygiene. The model can be state of the art. If your project graph is a mess, you still won’t ship. This is also exactly the kind of workflow I want to turn into a Terminal Skill. Not because a skill should hide the cloud setup behind magic, but because the debug order should not live only in someone’s memory. A useful vertex-ai-image-pipeline skill would give an agent a repeatable checklist: That is the broader idea behind Terminal Skills https://terminalskills.io : turn messy, real operational workflows into reusable agent skills. I will probably translate this article into a proper Terminal Skills use case next, because this is the kind of boring production workflow agents need more than another prompt template. If you’ve had to debug a broken AI pipeline recently, I’d genuinely love to hear what failed first for you.