Claude Codeto handle the heavy lifting of implementation, and for the first few sprints, it felt like magic. But the moment the codebase hit a certain level of complexity, the "vibe" shifted. I asked for a relatively simple patch to my authentication middleware—basically just adding a role-check—and the agent decided to refactor three other unrelated files to "optimize" the flow.
The result was a complete meltdown. I didn't realize the extent of the changes until I tried to boot the server and got slammed with a wall of errors.
Error: Cannot find module './utils/auth-helper'
at Module._load (node:internal/modules/cjs/:js, 934:34)
at Function.require (node:internal/modules/cjs/:js, 384:19)
at Object.<anonymous> (/src/middleware/auth.ts:12:1)
The diagnosis was frustrating. The agent hadn't just edited the logic; it had renamed a utility file and deleted the original without updating all the imports across the project. It essentially hallucinated a better file structure and applied it inconsistently. I spent two hours manually reverting commits because the agent's own "undo" function struggled to track the cross-file dependencies it had just shattered.
This is the danger of an AI workflow that lacks a rigorous test suite. When you're just "vibing" with the code, you assume the LLM understands the global state of the project, but it's really just operating on a sliding window of context. Once it loses the thread of where a specific function is exported, it starts guessing.
To stop this from happening again, I'm forcing a more disciplined prompt engineering approach. Instead of "Add X feature," I'm now using a strict step-by-step requirement:
-
Analyze the impact of the change on existing imports.
-
Propose the file changes in a plan first.
-
Execute the change without modifying any file not explicitly mentioned in the plan.
If you're doing a deep dive into LLM agents for development, the lesson here is that the more "autonomous" the agent is, the more likely it is to commit architectural suicide if you aren't watching the diffs in real-time. I'm now treating every agent-generated patch as a suspicious PR that needs a manual audit before it even touches the main branch.
For anyone else using these tools, are you seeing the same drift? I'm wondering if there's a way to lock the file structure so the agent can't just decide to rename things on a whim.
Next Why does the Transformer architecture look the way it does? →