# AI Coding Agents Can Pass Tests and Still Make the Wrong Decision > Source: > Published: 2026-08-13 15:53:34+00:00 A question I've been thinking about after discussing AI coding agents with several developers: Is passing the test suite enough to prove that an AI agent made the correct engineering decision? I don't think it is. And this isn't just a theoretical concern. Modern coding agents are increasingly working at the **repository level** rather than generating isolated code snippets. OpenAI's Codex documentation, for example, describes using repository-specific `AGENTS.md` instructions to tell the agent how to navigate a codebase, run tests, and follow project practices. Anthropic similarly describes Claude Code searching codebases, tracing dependencies, editing multiple files, and working with CI failures. ([OpenAI](https://openai.com/index/introducing-codex/?utm_source=chatgpt.com)) That changes what "correctness" means. A project starts with: ``` Architecture v1 API ↓ Service ↓ Database ``` An AI agent learns this structure and implements a new feature correctly. The tests pass. Then the architecture changes: ``` Architecture v2 API ↓ Event Bus ↓ Services ↓ Database ``` The same task is requested again. If the agent continues following the old architecture, its code might still: but still be **wrong for the current system**. This is the distinction I'm interested in: Code correctness ≠ Contextual correctness Traditional coding benchmarks generally provide: ``` Repository + Issue ↓ Agent ↓ Patch ↓ Tests / Evaluation ``` This is valuable. SWE-bench, for example, was designed around real GitHub issues and repositories, and OpenAI created SWE-bench Verified with human validation because benchmark quality itself affects what we conclude about model capability. ([OpenAI](https://openai.com/index/introducing-swe-bench-verified/?utm_source=chatgpt.com)) But there is another dimension worth testing: **What happens when the context changes?** Recent research is already moving in this direction. SWE-ContextBench evaluates whether coding agents can reuse relevant experience across related tasks, while SWE-Explore focuses specifically on repository exploration and context retrieval rather than treating the entire coding task as a single pass/fail outcome. ([arXiv](https://arxiv.org/abs/2602.08316?utm_source=chatgpt.com)) So I don't think the idea should be: "Replace existing coding benchmarks." Instead: Add controlled context-shift evaluations to them. Keep the model and task constant. Change only the relevant context. For example: ``` Architecture: REST → Service → Database Constraint: All database access must go through Repository classes. ``` Agent produces: ``` Controller ↓ Service ↓ Repository ↓ Database ``` ✅ Correct. Only change: ``` Architecture: REST → Event Bus → Service → Database ``` Now the expected implementation should change. If the agent still produces: ``` Controller ↓ Service ↓ Repository ``` then we have a measurable **context-adaptation failure**. We shouldn't reward an agent simply for changing its answer. Suppose we change something irrelevant: ``` README formatting ``` The architecture hasn't changed. The agent should ideally make the **same engineering decision**. So a useful benchmark should test both: ``` Relevant Context Change ↓ Decision SHOULD change ``` and: ``` Irrelevant Context Change ↓ Decision SHOULD remain stable ``` This gives us two complementary properties: Can the agent respond appropriately to relevant changes? Can the agent avoid unnecessary changes when the context is irrelevant? We could measure this quantitatively. For example: ``` Context Adaptation Rate = Correct decisions after relevant context changes / Total relevant context changes ``` And: ``` Context Stability = Unchanged decisions under irrelevant changes / Total irrelevant context changes ``` Then combine these with existing measures: ``` Agent Evaluation │ ├── Functional Correctness ├── Test Pass Rate ├── Constraint Adherence ├── Context Adaptation ├── Context Stability └── Repository Understanding ``` I'm not suggesting this is a finished benchmark methodology. It's a direction that I think is worth experimentally validating. The industry is already moving toward agents that operate over entire codebases. Anthropic's recent analysis of roughly **400,000 Claude Code sessions** describes agents being used for increasingly end-to-end software tasks, while engineers retain an important role in planning and directing the work. ([Anthropic](https://www.anthropic.com/research/claude-code-expertise?level=0&utm_source=chatgpt.com)) As agents receive more autonomy, the evaluation question changes. For a code completion system: "Is this code correct?" may be enough. For an agent modifying a long-lived production system: "Is this the correct decision given the current state, constraints, architecture, and history of the system?" becomes much more important. Maybe the next generation of coding-agent benchmarks shouldn't only measure: **Can the agent solve the task?** They should also measure: **Can the agent recognize when the task's surrounding reality has changed?** That could give us a more realistic picture of agent reliability. Not just: ``` Task → Code → Tests ``` but: ``` Task + Current Context + Constraints + Repository State + Previous Decisions ↓ Agent ↓ Decision ↓ Context-aware Evaluation ``` And importantly, this can be tested experimentally rather than treated as a vague concept. **What would you include in a context-shift benchmark first: architecture changes, security constraints, dependency changes, business requirements, or repository history?** `AGENTS.md` context (