A question I've been thinking about after discussing AI coding agents with several developers:
Is passing the test suite enough to prove that an AI agent made the correct engineering decision?
I don't think it is.
And this isn't just a theoretical concern.
Modern coding agents are increasingly working at the repository level rather than generating isolated code snippets. OpenAI's Codex documentation, for example, describes using repository-specific AGENTS.md
instructions to tell the agent how to navigate a codebase, run tests, and follow project practices. Anthropic similarly describes Claude Code searching codebases, tracing dependencies, editing multiple files, and working with CI failures. (OpenAI)
That changes what "correctness" means.
A project starts with:
Architecture v1
API
↓
Service
↓
Database
An AI agent learns this structure and implements a new feature correctly.
The tests pass.
Then the architecture changes:
Architecture v2
API
↓
Event Bus
↓
Services
↓
Database
The same task is requested again.
If the agent continues following the old architecture, its code might still:
but still be wrong for the current system.
This is the distinction I'm interested in:
Code correctness ≠ Contextual correctness
Traditional coding benchmarks generally provide:
Repository
+
Issue
↓
Agent
↓
Patch
↓
Tests / Evaluation
This is valuable.
SWE-bench, for example, was designed around real GitHub issues and repositories, and OpenAI created SWE-bench Verified with human validation because benchmark quality itself affects what we conclude about model capability. (OpenAI)
But there is another dimension worth testing:
What happens when the context changes?
Recent research is already moving in this direction.
SWE-ContextBench evaluates whether coding agents can reuse relevant experience across related tasks, while SWE-Explore focuses specifically on repository exploration and context retrieval rather than treating the entire coding task as a single pass/fail outcome. (arXiv)
So I don't think the idea should be:
"Replace existing coding benchmarks."
Instead:
Add controlled context-shift evaluations to them.
Keep the model and task constant.
Change only the relevant context.
For example:
Architecture:
REST → Service → Database
Constraint:
All database access must go through Repository classes.
Agent produces:
Controller
↓
Service
↓
Repository
↓
Database
✅ Correct.
Only change:
Architecture:
REST → Event Bus → Service → Database
Now the expected implementation should change.
If the agent still produces:
Controller
↓
Service
↓
Repository
then we have a measurable context-adaptation failure.
We shouldn't reward an agent simply for changing its answer.
Suppose we change something irrelevant:
README formatting
The architecture hasn't changed.
The agent should ideally make the same engineering decision.
So a useful benchmark should test both:
Relevant Context Change
↓
Decision SHOULD change
and:
Irrelevant Context Change
↓
Decision SHOULD remain stable
This gives us two complementary properties:
Can the agent respond appropriately to relevant changes?
Can the agent avoid unnecessary changes when the context is irrelevant?
We could measure this quantitatively.
For example:
Correct decisions after relevant context changes
/
Total relevant context changes
And:
Unchanged decisions under irrelevant changes
/
Total irrelevant context changes
Then combine these with existing measures:
Agent Evaluation
│
├── Functional Correctness
├── Test Pass Rate
├── Constraint Adherence
├── Context Adaptation
├── Context Stability
└── Repository Understanding
I'm not suggesting this is a finished benchmark methodology.
It's a direction that I think is worth experimentally validating.
The industry is already moving toward agents that operate over entire codebases.
Anthropic's recent analysis of roughly 400,000 Claude Code sessions describes agents being used for increasingly end-to-end software tasks, while engineers retain an important role in planning and directing the work. (Anthropic)
As agents receive more autonomy, the evaluation question changes.
For a code completion system:
"Is this code correct?"
may be enough.
For an agent modifying a long-lived production system:
"Is this the correct decision given the current state, constraints, architecture, and history of the system?"
becomes much more important.
Maybe the next generation of coding-agent benchmarks shouldn't only measure:
Can the agent solve the task?
They should also measure:
Can the agent recognize when the task's surrounding reality has changed?
That could give us a more realistic picture of agent reliability.
Not just:
Task → Code → Tests
but:
Task
+
Current Context
+
Constraints
+
Repository State
+
Previous Decisions
↓
Agent
↓
Decision
↓
Context-aware Evaluation
And importantly, this can be tested experimentally rather than treated as a vague concept.
What would you include in a context-shift benchmark first: architecture changes, security constraints, dependency changes, business requirements, or repository history?
AGENTS.md
context (