{"slug": "ai-coding-agents-can-pass-tests-and-still-make-the-wrong-decision", "title": "AI Coding Agents Can Pass Tests and Still Make the Wrong Decision", "summary": "An engineer argues that passing test suites is insufficient to prove an AI coding agent made the correct engineering decision, highlighting the need for context-adaptation benchmarks. The developer proposes evaluating agents on their ability to change decisions when relevant context changes and remain stable when it doesn't, introducing metrics like Context Adaptation Rate and Context Stability.", "body_md": "A question I've been thinking about after discussing AI coding agents with several developers:\n\nIs passing the test suite enough to prove that an AI agent made the correct engineering decision?\n\nI don't think it is.\n\nAnd this isn't just a theoretical concern.\n\nModern coding agents are increasingly working at the **repository level** rather than generating isolated code snippets. OpenAI's Codex documentation, for example, describes using repository-specific `AGENTS.md`\n\ninstructions to tell the agent how to navigate a codebase, run tests, and follow project practices. Anthropic similarly describes Claude Code searching codebases, tracing dependencies, editing multiple files, and working with CI failures. ([OpenAI](https://openai.com/index/introducing-codex/?utm_source=chatgpt.com))\n\nThat changes what \"correctness\" means.\n\nA project starts with:\n\n```\nArchitecture v1\n\nAPI\n ↓\nService\n ↓\nDatabase\n```\n\nAn AI agent learns this structure and implements a new feature correctly.\n\nThe tests pass.\n\nThen the architecture changes:\n\n```\nArchitecture v2\n\nAPI\n ↓\nEvent Bus\n ↓\nServices\n ↓\nDatabase\n```\n\nThe same task is requested again.\n\nIf the agent continues following the old architecture, its code might still:\n\nbut still be **wrong for the current system**.\n\nThis is the distinction I'm interested in:\n\nCode correctness ≠ Contextual correctness\n\nTraditional coding benchmarks generally provide:\n\n```\nRepository\n+\nIssue\n↓\nAgent\n↓\nPatch\n↓\nTests / Evaluation\n```\n\nThis is valuable.\n\nSWE-bench, for example, was designed around real GitHub issues and repositories, and OpenAI created SWE-bench Verified with human validation because benchmark quality itself affects what we conclude about model capability. ([OpenAI](https://openai.com/index/introducing-swe-bench-verified/?utm_source=chatgpt.com))\n\nBut there is another dimension worth testing:\n\n**What happens when the context changes?**\n\nRecent research is already moving in this direction.\n\nSWE-ContextBench evaluates whether coding agents can reuse relevant experience across related tasks, while SWE-Explore focuses specifically on repository exploration and context retrieval rather than treating the entire coding task as a single pass/fail outcome. ([arXiv](https://arxiv.org/abs/2602.08316?utm_source=chatgpt.com))\n\nSo I don't think the idea should be:\n\n\"Replace existing coding benchmarks.\"\n\nInstead:\n\nAdd controlled context-shift evaluations to them.\n\nKeep the model and task constant.\n\nChange only the relevant context.\n\nFor example:\n\n```\nArchitecture:\nREST → Service → Database\n\nConstraint:\nAll database access must go through Repository classes.\n```\n\nAgent produces:\n\n```\nController\n   ↓\nService\n   ↓\nRepository\n   ↓\nDatabase\n```\n\n✅ Correct.\n\nOnly change:\n\n```\nArchitecture:\nREST → Event Bus → Service → Database\n```\n\nNow the expected implementation should change.\n\nIf the agent still produces:\n\n```\nController\n   ↓\nService\n   ↓\nRepository\n```\n\nthen we have a measurable **context-adaptation failure**.\n\nWe shouldn't reward an agent simply for changing its answer.\n\nSuppose we change something irrelevant:\n\n```\nREADME formatting\n```\n\nThe architecture hasn't changed.\n\nThe agent should ideally make the **same engineering decision**.\n\nSo a useful benchmark should test both:\n\n```\nRelevant Context Change\n        ↓\nDecision SHOULD change\n```\n\nand:\n\n```\nIrrelevant Context Change\n        ↓\nDecision SHOULD remain stable\n```\n\nThis gives us two complementary properties:\n\nCan the agent respond appropriately to relevant changes?\n\nCan the agent avoid unnecessary changes when the context is irrelevant?\n\nWe could measure this quantitatively.\n\nFor example:\n\n```\nContext Adaptation Rate\n=\nCorrect decisions after relevant context changes\n/\nTotal relevant context changes\n```\n\nAnd:\n\n```\nContext Stability\n=\nUnchanged decisions under irrelevant changes\n/\nTotal irrelevant context changes\n```\n\nThen combine these with existing measures:\n\n```\nAgent Evaluation\n│\n├── Functional Correctness\n├── Test Pass Rate\n├── Constraint Adherence\n├── Context Adaptation\n├── Context Stability\n└── Repository Understanding\n```\n\nI'm not suggesting this is a finished benchmark methodology.\n\nIt's a direction that I think is worth experimentally validating.\n\nThe industry is already moving toward agents that operate over entire codebases.\n\nAnthropic's recent analysis of roughly **400,000 Claude Code sessions** describes agents being used for increasingly end-to-end software tasks, while engineers retain an important role in planning and directing the work. ([Anthropic](https://www.anthropic.com/research/claude-code-expertise?level=0&utm_source=chatgpt.com))\n\nAs agents receive more autonomy, the evaluation question changes.\n\nFor a code completion system:\n\n\"Is this code correct?\"\n\nmay be enough.\n\nFor an agent modifying a long-lived production system:\n\n\"Is this the correct decision given the current state, constraints, architecture, and history of the system?\"\n\nbecomes much more important.\n\nMaybe the next generation of coding-agent benchmarks shouldn't only measure:\n\n**Can the agent solve the task?**\n\nThey should also measure:\n\n**Can the agent recognize when the task's surrounding reality has changed?**\n\nThat could give us a more realistic picture of agent reliability.\n\nNot just:\n\n```\nTask → Code → Tests\n```\n\nbut:\n\n```\nTask\n +\nCurrent Context\n +\nConstraints\n +\nRepository State\n +\nPrevious Decisions\n        ↓\n      Agent\n        ↓\n     Decision\n        ↓\nContext-aware Evaluation\n```\n\nAnd importantly, this can be tested experimentally rather than treated as a vague concept.\n\n**What would you include in a context-shift benchmark first: architecture changes, security constraints, dependency changes, business requirements, or repository history?**\n\n`AGENTS.md`\n\ncontext (", "url": "https://wpnews.pro/news/ai-coding-agents-can-pass-tests-and-still-make-the-wrong-decision", "canonical_source": "https://dev.to/probal_dhali_f7d15eac866a/-ai-coding-agents-can-pass-tests-and-still-make-the-wrong-decision-3hf7", "published_at": "2026-08-13 15:53:34+00:00", "updated_at": "2026-08-13 16:20:21.934416+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "developer-tools"], "entities": ["OpenAI", "Anthropic", "SWE-bench", "SWE-ContextBench", "SWE-Explore"], "alternates": {"html": "https://wpnews.pro/news/ai-coding-agents-can-pass-tests-and-still-make-the-wrong-decision", "markdown": "https://wpnews.pro/news/ai-coding-agents-can-pass-tests-and-still-make-the-wrong-decision.md", "text": "https://wpnews.pro/news/ai-coding-agents-can-pass-tests-and-still-make-the-wrong-decision.txt", "jsonld": "https://wpnews.pro/news/ai-coding-agents-can-pass-tests-and-still-make-the-wrong-decision.jsonld"}}