AI coding agents are getting better at writing code.
But I think we are approaching a more difficult question:
How do we know that an AI agent made the right engineering decision for the current state of a software system?
Passing tests is important.
But passing tests alone does not necessarily tell us whether an agent understood:
This becomes particularly important as AI systems move from generating isolated code snippets toward modifying real repositories.
Consider a simple example.
A project initially has:
Architecture v1
API
↓
Service
↓
Database
An AI agent is asked to add a feature.
It studies the repository, follows the existing pattern, writes the code, and all tests pass.
Then the architecture changes:
Architecture v2
API
↓
Event Bus
↓
Service
↓
Database
The same task is requested again.
If the agent still generates code based on the old architecture, the implementation may be:
✓ Valid syntax
✓ Compiles
✓ Existing tests pass
✗ Violates current architecture
✗ Ignores current constraints
So we have an important distinction:
Functional Correctness
≠
Contextual Correctness
≠
System-Level Correctness
This is the problem I want to explore.
This isn't simply speculation about future AI systems.
Modern coding agents already depend on repository-level context.
OpenAI's documentation for Codex recommends using persistent repository instructions such as AGENTS.md
for naming conventions, business logic, known quirks, dependencies, and other information that may not be inferable directly from code. It also recommends providing file paths, component names, diffs, and documentation when describing tasks.
OpenAI has also described a broader approach where repository knowledge becomes a structured source of truth rather than one giant instruction document, explicitly noting that context management is one of the biggest challenges for agents working on large and complex tasks.
That leads to an interesting conclusion:
If context materially affects agent performance, context should also become part of agent evaluation.
SWE-bench was created to evaluate AI systems on real software-engineering issues from GitHub repositories.
The agent receives a repository and an issue, modifies the code, and is evaluated using tests. SWE-bench Verified was later created as a human-validated subset after OpenAI and the SWE-bench authors found problems with some benchmark tasks. 500 tasks were selected after professional developers screened the data.
But benchmark methodology itself is evolving.
In February 2026, OpenAI reported that SWE-bench Verified had become increasingly contaminated and recommended newer evaluations such as SWE-bench Pro.
In July 2026, OpenAI also reported that its audit of SWE-bench Pro found widespread task-quality problems and estimated roughly 30% of tasks were broken.
That matters because it demonstrates a broader lesson:
Evaluating AI systems is itself an engineering problem.
A benchmark can produce a number without necessarily producing a reliable measurement.
This is also not an isolated idea.
A 2026 research benchmark called SWE-ContextBench specifically investigates whether coding agents can reuse relevant experience across related software-engineering tasks.
The benchmark augments SWE-bench Lite with related tasks derived from dependency and reference relationships between GitHub issues and pull requests. It evaluates prediction accuracy, time efficiency, and cost efficiency. The authors report that appropriately selected summarized experience can improve resolution accuracy while reducing runtime and token cost, whereas poorly selected experience can provide limited or negative benefits.
That suggests something important:
More context
≠
Better result
The real question is:
Relevant context
+
Correct retrieval
+
Correct interpretation
↓
Better decision
I propose thinking about agent evaluation as a multi-dimensional problem.
Instead of:
Task
↓
Agent
↓
Code
↓
Tests
↓
Pass / Fail
we could evaluate:
┌───────────────┐
│ TASK │
└───────┬───────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
Repository Constraints History
│ │ │
└───────────┼───────────┘
▼
AI CODING AGENT
│
▼
DECISION
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Code Quality Constraints Context Fit
│ │ │
└──────────────┼──────────────┘
▼
Final Score
This is the idea I find particularly interesting.
Keep the:
Model
Task
Repository
as constant as possible.
Then change one meaningful part of the context.
For example:
Database:
PostgreSQL
Architecture:
Repository pattern
Constraint:
All database access must go through repositories.
The agent produces:
Controller
↓
Service
↓
Repository
↓
PostgreSQL
Good.
Change one relevant constraint:
Database:
PostgreSQL
Architecture:
Event-driven
Constraint:
Services must communicate through events.
Now the appropriate implementation should change.
If the agent continues producing the old architecture, we can measure a context adaptation failure.
We shouldn't reward an agent merely for changing its answer.
Suppose we change something irrelevant:
README formatting
The architecture hasn't changed.
The agent should ideally make the same engineering decision.
So:
Relevant context changes
↓
Decision SHOULD change
while:
Irrelevant context changes
↓
Decision SHOULD remain stable
This gives us two useful properties.
Does the agent react when relevant context changes?
Does the agent remain stable when irrelevant context changes?
A practical benchmark could use paired or grouped scenarios:
Task T
Context C1
↓
Agent
↓
Decision D1
Task T
Context C2
↓
Agent
↓
Decision D2
Where:
C1 → C2
contains a controlled change.
Then evaluate:
Was the change relevant?
↓
Should the decision change?
↓
Did the agent change?
↓
Was the new decision correct?
And separately:
Was the context change irrelevant?
↓
Should the decision remain stable?
↓
Did the agent unnecessarily change?
I wouldn't claim these are established industry-standard metrics. They are a proposed framework that would need experimental validation.
Did the implementation satisfy the task?
Did the implementation respect explicit constraints?
When relevant context changed, how often did the agent make the appropriate change?
When irrelevant context changed, how often did the agent preserve the appropriate decision?
Does the change follow the project's established architecture and conventions?
Did the change break previously working behavior?
How much context did the agent need to retrieve to make the correct decision?
This could eventually produce something like:
Agent Reliability Score
│
├── Functional Correctness
├── Constraint Adherence
├── Context Adaptation
├── Context Stability
├── Repository Consistency
├── Regression Resistance
└── Context Efficiency
A common reaction might be:
"Just give the model the entire repository."
But that's not necessarily a solution.
OpenAI's own engineering discussion around Codex describes the problem with extremely large instruction documents: context is limited, important information can be crowded out, stale instructions can accumulate, and humans may stop maintaining them. Their approach is instead to use a concise map pointing toward deeper sources of truth.
So the problem isn't simply:
How much context?
It is:
Which context?
When?
From where?
How current?
How reliable?
How relevant?
That is a much more interesting systems problem.
I think repository context should be treated as something that changes over time:
Initial Decision
↓
Implementation
↓
New Requirement
↓
Architecture Change
↓
Dependency Change
↓
Security Change
↓
New Decision
An agent working on a long-lived repository therefore needs something closer to:
Current State
+
Historical Decisions
+
Active Constraints
+
Repository Structure
+
Relevant Documentation
rather than simply:
Prompt + Code
Two implementations can be functionally equivalent while only one fits the project.
For example:
cache_result()
versus:
await cache_result()
Both might pass a narrow test.
But the correct choice could depend on:
The code itself doesn't always contain the complete explanation.
Sometimes the most important information is why the code was designed that way.
Traditional benchmark thinking often looks like:
Fixed Task
↓
Fixed Dataset
↓
Fixed Evaluation
↓
Score
But real repositories look more like:
Task
↓
Repository evolves
↓
Requirements change
↓
Dependencies change
↓
Architecture changes
↓
Security constraints change
↓
Agent receives new task
Therefore, a future benchmark could intentionally introduce controlled environmental changes.
For example:
Version 1
↓
Agent decision
Version 2
↓
Architecture changed
Version 3
↓
Security policy changed
Version 4
↓
Dependency changed
Then measure whether the agent adapts correctly.
One simple way to visualize the experiment:
| Decision Should Stay Same | Decision Should Change | |
|---|---|---|
| Agent stays same | ||
| ✅ Stable | ❌ Adaptation failure | |
| Agent changes | ||
| ❌ Instability | ✅ Adaptation success |
This is interesting because it separates two failure modes that ordinary pass/fail evaluation can hide.
The question isn't:
"Can AI write code?"
We're already measuring that.
The more difficult question is:
"Can an AI agent maintain correct engineering judgment as the software environment changes?"
That includes:
Architecture
Requirements
Dependencies
Security
Performance
Business Rules
Repository History
Team Conventions
This is closer to how real software development works.
When investigating a complex AI engineering problem, I don't think one reasoning style is enough.
Sometimes we need a diagram.
Sometimes a benchmark.
Sometimes a root-cause analysis.
Sometimes a comparison.
Sometimes a threat model.
Sometimes a timeline.
Sometimes a first-principles explanation.
So I compiled a reusable set of 100 visual-thinking, explanation, analysis, and strategy lenses.
These aren't claims about AI capability. They are ways to structure thinking and communicate technical problems.
| # | Shortcut | Lens |
|---|---|---|
| 1 | /handwritten |
|
| Notebook-style handwritten notes | ||
| 2 | /visualize |
|
| Turn ideas into visual explanations | ||
| 3 | /stickynotes |
|
| One idea per sticky note | ||
| 4 | /infographic |
|
| Infographic layout | ||
| 5 | /diagram |
|
| Draw a concept diagram | ||
| 6 | /flowchart |
|
| Step-by-step flowchart | ||
| 7 | /mindmap |
|
| Create a mind map | ||
| 8 | /xray |
|
| Show internal structure | ||
| 9 | /blueprint |
|
| Technical blueprint | ||
| 10 | /explodedview |
|
| Break object into components | ||
| 11 | /thenvsnow |
|
| Compare past vs present | ||
| 12 | /timeline |
|
| Chronological timeline | ||
| 13 | /beforeafter |
|
| Transformation comparison | ||
| 14 | /cutaway |
|
| Cutaway illustration | ||
| 15 | /anatomy |
|
| Explain all parts | ||
| 16 | /layers |
|
| Layer-by-layer architecture | ||
| 17 | /ecosystem |
|
| Show all connected players | ||
| 18 | /journey |
|
| Show end-to-end journey | ||
| 19 | /process |
|
| Explain a complete process | ||
| 20 | /cycle |
|
| Visualize recurring cycles | ||
| 21 | /roadmap |
|
| Learning or execution roadmap | ||
| 22 | /dashboard |
|
| Dashboard with KPIs | ||
| 23 | /comparison |
|
| Side-by-side comparison | ||
| 24 | /versus |
|
| Head-to-head comparison | ||
| 25 | /scale |
|
| Compare sizes visually | ||
| 26 | /evolution |
|
| Show evolution over time | ||
| 27 | /future |
|
| Imagine future scenarios | ||
| 28 | /inside |
|
| Reveal inner workings | ||
| 29 | /microscopic |
|
| Zoom into microscopic detail | ||
| 30 | /macroscopic |
|
| Zoom out to system level | ||
| 31 | /crosssection |
|
| Cross-sectional illustration | ||
| 32 | /map |
|
| Geographic or conceptual map | ||
| 33 | /heatmap |
|
| Show intensity | ||
| 34 | /network |
|
| Show relationships | ||
| 35 | /architecture |
|
| Software/system architecture | ||
| 36 | /wireframe |
|
| Website/app layout | ||
| 37 | /mockup |
|
| Realistic product preview | ||
| 38 | /prototype |
|
| Early product concept | ||
| 39 | /schematic |
|
| Simple technical schematic | ||
| 40 | /isometric |
|
| 3D isometric illustration | ||
| 41 | /birdseye |
|
| Top-down view | ||
| 42 | /360view |
|
| All-angle visualization | ||
| 43 | /storyboard |
|
| Scene-by-scene explanation | ||
| 44 | /comic |
|
| Explain through comic panels | ||
| 45 | /poster |
|
| Poster design | ||
| 46 | /cover |
|
| Book/report cover | ||
| 47 | /adcreative |
|
| Advertising concept | ||
| 48 | /thumbnail |
|
| YouTube thumbnail concept | ||
| 49 | /carousel |
|
| Instagram/LinkedIn carousel | ||
| 50 | /socialvisual |
|
| Social media graphic | ||
| 51 | /quotevisual |
|
| Quote as shareable visual | ||
| 52 | /eli5 |
|
| Explain simply | ||
| 53 | /expert |
|
| Expert-level explanation | ||
| 54 | /firstprinciples |
|
| Break down to fundamentals | ||
| 55 | /deepdive |
|
| Comprehensive explanation | ||
| 56 | /simplify |
|
| Simplify difficult content | ||
| 57 | /analogy |
|
| Explain through analogy | ||
| 58 | /socratic |
|
| Teach through questions | ||
| 59 | /teachme |
|
| Structured tutoring | ||
| 60 | /cheatsheet |
|
| Quick-reference notes | ||
| 61 | /flashcards |
|
| Study flashcards | ||
| 62 | /quiz |
|
| Generate a quiz | ||
| 63 | /viva |
|
| Viva preparation | ||
| 64 | /interview |
|
| Mock interview | ||
| 65 | /devilsadvocate |
|
| Challenge assumptions | ||
| 66 | /factcheck |
|
| Verify claims | ||
| 67 | /mythvsfact |
|
| Separate myths from facts | ||
| 68 | /proscons |
|
| Advantages vs disadvantages | ||
| 69 | /swot |
|
| SWOT analysis | ||
| 70 | /pestle |
|
| PESTLE analysis | ||
| 71 | /fiveforces |
|
| Porter's Five Forces | ||
| 72 | /rootcause |
|
| Find root cause | ||
| 73 | /fivewhys |
|
| Five Whys analysis | ||
| 74 | /decisionmatrix |
|
| Weighted decision matrix | ||
| 75 | /scenario |
|
| Scenario planning | ||
| 76 | /simulate |
|
| Simulation exercise | ||
| 77 | /roleplay |
|
| Assume an expert role | ||
| 78 | /consultant |
|
| Consulting-style advice | ||
| 79 | /executivebrief |
|
| Executive summary | ||
| 80 | /insights |
|
| Extract insights | ||
| 81 | /recommendations |
|
| Provide recommendations | ||
| 82 | /prioritize |
|
| Rank by priority | ||
| 83 | /benchmark |
|
| Benchmark comparison | ||
| 84 | /marketmap |
|
| Industry landscape | ||
| 85 | /strategy |
|
| Strategic planning | ||
| 86 | /businessmodel |
|
| Business model explanation | ||
| 87 | /pitch |
|
| Investor/startup pitch | ||
| 88 | /investor |
|
| Investor perspective | ||
| 89 | /redteam |
|
| Stress-test a plan | ||
| 90 | /premortem |
|
| Assume failure and analyze why | ||
| 91 | /reverseengineer |
|
| Break down success | ||
| 92 | /promptengineer |
|
| Optimize prompts | ||
| 93 | /research |
|
| Structured research | ||
| 94 | /sources |
|
| Find reliable sources | ||
| 95 | /summarize |
|
| Summarize content | ||
| 96 | /extract |
|
| Extract key information | ||
| 97 | /table |
|
| Convert into a table | ||
| 98 | /presentation |
|
| Presentation outline | ||
| 99 | /dashboardanalysis |
|
| Analyze dashboards | ||
| 100 | /actionplan |
|
| Create step-by-step action plan |
These 100 lenses are not 100 claims that an AI model is more intelligent when using them.
They are simply structured ways of looking at a problem.
For AI-agent research, different lenses can answer different questions:
/architecture
↓
What is the system structure?
/xray
↓
What is happening internally?
/timeline
↓
How did the system change?
/thenvsnow
↓
What changed between versions?
/benchmark
↓
How should we measure it?
/factcheck
↓
Which claims have evidence?
/redteam
↓
How can the evaluation fail?
/rootcause
↓
Why did the agent fail?
/decisionmatrix
↓
Which approach is better?
/actionplan
↓
What should we build next?
This is especially useful when researching complex AI systems because no single representation captures the entire problem.
If I were turning this idea into an actual research experiment, I'd start small.
Create 50–100 repository-level tasks.
For each task, create controlled variants:
Architecture change
Requirement change
Security constraint change
Dependency change
Performance constraint change
Documentation change
Irrelevant formatting change
Run:
Same Model
Same Task
Different Context
Then measure:
Functional correctness
Context adaptation
Context stability
Constraint adherence
Regression
Token usage
Runtime
For ambiguous cases, use experienced developers to verify whether the changed decision was actually appropriate.
This is important because benchmark design itself can introduce errors. OpenAI's SWE-bench work demonstrates why human validation and benchmark auditing matter when interpreting agent performance.
My current hypothesis is:
A reliable coding agent should not simply produce correct code. It should produce decisions that are appropriate for the current context, adapt when relevant context changes, and remain stable when irrelevant context changes.
That's a much stronger definition of reliability.
And importantly, it is something we can attempt to measure.
AI coding agents are moving from:
Code Completion
toward:
Software Engineering Agents
As that transition happens, our evaluation methods need to evolve too.
The future benchmark may not simply ask:
“Did the code pass?”
It may need to ask:
“Did the agent understand the current system well enough to make the right engineering decision?”
That is the problem I find most interesting.
And I don't think we have completely solved it yet.
What would you add to a context-aware coding-agent benchmark?
Architecture changes?
Security constraints?
Dependency changes?
Business requirements?
Repository history?
I'd genuinely like to hear how other developers would design it.
AGENTS.md
, task specification, and development-environment guidance.