{"slug": "why-i-stopped-trusting-exit-code-0-from-ai-coding-agents", "title": "Why I stopped trusting “exit code 0” from AI coding agents", "summary": "A developer built ELY Agent Input Preflight, an open-source CLI that independently verifies AI coding agent runs instead of trusting the agent's exit code 0. The tool checks required project inputs before a run, starts Codex in a read-only sandbox unless write access is explicitly enabled, snapshots files, and runs its own Postflight verification (such as a Python unittest suite) after the agent exits, returning VERIFIED, FAILED, or NEEDS_REVIEW. The developer cites a Codex run that exited 0 while tests failed inside its sandbox, but passed under ELY's independent verification.", "body_md": "AI coding agents are getting much better.\n\nThey can inspect repositories, change multiple files, run commands, write tests, and sometimes complete work that would have taken a developer hours.\n\nBut I kept running into the same problem:\n\nthe agent would finish, return exit code 0, and say that everything was successful.\n\nSometimes it really was.\n\nSometimes it wasn't.\n\nThat made me realize that I did not only need a better coding agent.\n\nI needed an independent way to verify what the agent had actually done.\n\nWhen an AI coding agent works on a project, several things can go wrong:\n\nThe stronger coding agents become, the more important this problem becomes.\n\nIf an agent changes 10 or 20 files, I do not want to manually inspect every line after every run.\n\nAnd if agents become even more autonomous, manual verification becomes even less practical.\n\nThat is the problem I wanted to solve.\n\nI built a small open-source CLI called **ELY Agent Input Preflight**.\n\n👉 **Try ELY Agent Input Preflight on GitHub:**\n\n[https://github.com/Golovenkov79/ely-agent-input-preflight](https://github.com/Golovenkov79/ely-agent-input-preflight)\n\nThe idea is simple:\n\nThe important part is this:\n\n**an agent exit code of 0 is not automatically treated as success.**\n\nELY can return:\n\n`VERIFIED`` FAILED``NEEDS_REVIEW`\nSo the agent's own result is only one signal.\n\nIt is not the final authority.\n\nThe first layer is Preflight.\n\nBefore the coding agent runs, ELY checks whether the project has the inputs it is supposed to have.\n\nThat can include things such as:\n\nIf something critical is missing, ELY can stop the run before the coding agent starts.\n\nThat sounds simple, but it avoids wasting an expensive agent run just to discover later that the required context was incomplete.\n\nELY also detects the type of project it is looking at.\n\nThe current release includes project detection for:\n\nBased on that detection, ELY can select matching built-in guidance before preparing the handoff to the coding agent.\n\nThe goal is not to replace the agent.\n\nThe goal is to make sure the agent starts with better context and clearer rules.\n\nOne design decision was especially important to me:\n\n**the coding agent should not get write access by default.**\n\nWith the current Codex integration, ELY starts the agent in an explicit read-only sandbox unless write access is deliberately enabled.\n\nIf I only want an agent to inspect a project, I do not want it silently modifying files.\n\nWrite access has to be explicitly requested.\n\nThat gives me a much clearer separation between:\n\nELY also takes a snapshot of project files before an agent run.\n\nAfter the agent exits, it compares the workspace again.\n\nThis allows ELY to detect:\n\nThat matters especially for read-only runs.\n\nIf a supposedly read-only agent run changes protected project files, ELY can treat the result as a failure.\n\nThis is independent of what the agent claims it did.\n\nDuring development I had a Codex run where the agent completed with exit code 0.\n\nInside its sandbox, however, many tests could not run correctly because temporary writable storage was unavailable.\n\nThe agent still completed its inspection and reported the limitation.\n\nELY did not simply trust the agent result.\n\nAfter Codex exited, ELY ran its own Postflight verification outside the agent sandbox and independently checked the project.\n\nThe Postflight verifier was able to run the project's tests normally and verify the state independently.\n\nThat was the moment when the idea became useful to me.\n\nThe interesting part was not that the coding agent was \"bad\".\n\nIt was that the agent and the verifier were operating under different conditions.\n\nA successful agent run and a successfully verified project are not always the same thing.\n\nAfter the coding agent exits, ELY performs its own verification step.\n\nFor Python projects, the current built-in verifier can run the project's unittest suite when a `tests` directory is available.\n\nThe final result can then be classified as:\n\nThe independent verification passed.\n\nThe agent failed, the project changed when it should not have, the Preflight checks became blocked, or the independent verifier failed.\n\nELY does not currently have enough automated verification for that project type.\n\nI prefer this to pretending that every run can be automatically proven correct.\n\nSometimes the right answer really is:\n\n**this needs a human review.**\n\n``` php\ntext\nPreflight\n  -> Project detection\n  -> Skill selection\n  -> Agent handoff\n  -> Codex\n  -> File integrity check\n  -> Independent Postflight verification\n  -> Run history\n```\n\n", "url": "https://wpnews.pro/news/why-i-stopped-trusting-exit-code-0-from-ai-coding-agents", "canonical_source": "https://dev.to/__f61d6e248/why-i-stopped-trusting-exit-code-0-from-ai-coding-agents-1aaa", "published_at": "2026-09-28 05:55:03+00:00", "updated_at": "2026-09-28 06:18:23.436179+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["ELY Agent Input Preflight", "Codex", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-i-stopped-trusting-exit-code-0-from-ai-coding-agents", "markdown": "https://wpnews.pro/news/why-i-stopped-trusting-exit-code-0-from-ai-coding-agents.md", "text": "https://wpnews.pro/news/why-i-stopped-trusting-exit-code-0-from-ai-coding-agents.txt", "jsonld": "https://wpnews.pro/news/why-i-stopped-trusting-exit-code-0-from-ai-coding-agents.jsonld"}}