AI Coding Agents Can Pass Tests and Still Make the Wrong Decision

An engineer argues that passing test suites is insufficient to prove an AI coding agent made the correct engineering decision, highlighting the need for context-adaptation benchmarks. The developer proposes evaluating agents on their ability to change decisions when relevant context changes and remain stable when it doesn't, introducing metrics like Context Adaptation Rate and Context Stability.

A question I've been thinking about after discussing AI coding agents with several developers: Is passing the test suite enough to prove that an AI agent made the correct engineering decision? I don't think it is. And this isn't just a theoretical concern. Modern coding agents are increasingly working at the repository level rather than generating isolated code snippets. OpenAI's Codex documentation, for example, describes using repository-specific AGENTS.md instructions to tell the agent how to navigate a codebase, run tests, and follow project practices. Anthropic similarly describes Claude Code searching codebases, tracing dependencies, editing multiple files, and working with CI failures. OpenAI https://openai.com/index/introducing-codex/?utm source=chatgpt.com That changes what "correctness" means. A project starts with: Architecture v1 API ↓ Service ↓ Database An AI agent learns this structure and implements a new feature correctly. The tests pass. Then the architecture changes: Architecture v2 API ↓ Event Bus ↓ Services ↓ Database The same task is requested again. If the agent continues following the old architecture, its code might still: but still be wrong for the current system . This is the distinction I'm interested in: Code correctness ≠ Contextual correctness Traditional coding benchmarks generally provide: Repository + Issue ↓ Agent ↓ Patch ↓ Tests / Evaluation This is valuable. SWE-bench, for example, was designed around real GitHub issues and repositories, and OpenAI created SWE-bench Verified with human validation because benchmark quality itself affects what we conclude about model capability. OpenAI https://openai.com/index/introducing-swe-bench-verified/?utm source=chatgpt.com But there is another dimension worth testing: What happens when the context changes? Recent research is already moving in this direction. SWE-ContextBench evaluates whether coding agents can reuse relevant experience across related tasks, while SWE-Explore focuses specifically on repository exploration and context retrieval rather than treating the entire coding task as a single pass/fail outcome. arXiv https://arxiv.org/abs/2602.08316?utm source=chatgpt.com So I don't think the idea should be: "Replace existing coding benchmarks." Instead: Add controlled context-shift evaluations to them. Keep the model and task constant. Change only the relevant context. For example: Architecture: REST → Service → Database Constraint: All database access must go through Repository classes. Agent produces: Controller ↓ Service ↓ Repository ↓ Database ✅ Correct. Only change: Architecture: REST → Event Bus → Service → Database Now the expected implementation should change. If the agent still produces: Controller ↓ Service ↓ Repository then we have a measurable context-adaptation failure . We shouldn't reward an agent simply for changing its answer. Suppose we change something irrelevant: README formatting The architecture hasn't changed. The agent should ideally make the same engineering decision . So a useful benchmark should test both: Relevant Context Change ↓ Decision SHOULD change and: Irrelevant Context Change ↓ Decision SHOULD remain stable This gives us two complementary properties: Can the agent respond appropriately to relevant changes? Can the agent avoid unnecessary changes when the context is irrelevant? We could measure this quantitatively. For example: Context Adaptation Rate = Correct decisions after relevant context changes / Total relevant context changes And: Context Stability = Unchanged decisions under irrelevant changes / Total irrelevant context changes Then combine these with existing measures: Agent Evaluation │ ├── Functional Correctness ├── Test Pass Rate ├── Constraint Adherence ├── Context Adaptation ├── Context Stability └── Repository Understanding I'm not suggesting this is a finished benchmark methodology. It's a direction that I think is worth experimentally validating. The industry is already moving toward agents that operate over entire codebases. Anthropic's recent analysis of roughly 400,000 Claude Code sessions describes agents being used for increasingly end-to-end software tasks, while engineers retain an important role in planning and directing the work. Anthropic https://www.anthropic.com/research/claude-code-expertise?level=0&utm source=chatgpt.com As agents receive more autonomy, the evaluation question changes. For a code completion system: "Is this code correct?" may be enough. For an agent modifying a long-lived production system: "Is this the correct decision given the current state, constraints, architecture, and history of the system?" becomes much more important. Maybe the next generation of coding-agent benchmarks shouldn't only measure: Can the agent solve the task? They should also measure: Can the agent recognize when the task's surrounding reality has changed? That could give us a more realistic picture of agent reliability. Not just: Task → Code → Tests but: Task + Current Context + Constraints + Repository State + Previous Decisions ↓ Agent ↓ Decision ↓ Context-aware Evaluation And importantly, this can be tested experimentally rather than treated as a vague concept. What would you include in a context-shift benchmark first: architecture changes, security constraints, dependency changes, business requirements, or repository history? AGENTS.md context