{"slug": "beyond-passing-tests-a-100-lens-framework-for-evaluating-context-aware-ai-coding", "title": "Beyond Passing Tests: A 100-Lens Framework for Evaluating Context-Aware AI Coding Agents 🤖", "summary": "A developer proposes a 100-lens framework for evaluating context-aware AI coding agents, arguing that passing tests alone is insufficient to measure whether an agent understood the current state of a software system. The framework distinguishes functional, contextual, and system-level correctness, and highlights the importance of repository-level context in agent performance. The developer cites evolving benchmarks like SWE-bench Pro and SWE-ContextBench to illustrate that evaluation itself is an engineering challenge.", "body_md": "AI coding agents are getting better at writing code.\n\nBut I think we are approaching a more difficult question:\n\nHow do we know that an AI agent made the right engineering decision for the current state of a software system?\n\nPassing tests is important.\n\nBut passing tests alone does not necessarily tell us whether an agent understood:\n\nThis becomes particularly important as AI systems move from generating isolated code snippets toward modifying real repositories.\n\nConsider a simple example.\n\nA project initially has:\n\n```\nArchitecture v1\n\nAPI\n ↓\nService\n ↓\nDatabase\n```\n\nAn AI agent is asked to add a feature.\n\nIt studies the repository, follows the existing pattern, writes the code, and all tests pass.\n\nThen the architecture changes:\n\n```\nArchitecture v2\n\nAPI\n ↓\nEvent Bus\n ↓\nService\n ↓\nDatabase\n```\n\nThe same task is requested again.\n\nIf the agent still generates code based on the old architecture, the implementation may be:\n\n```\n✓ Valid syntax\n✓ Compiles\n✓ Existing tests pass\n✗ Violates current architecture\n✗ Ignores current constraints\n```\n\nSo we have an important distinction:\n\n```\nFunctional Correctness\n        ≠\nContextual Correctness\n        ≠\nSystem-Level Correctness\n```\n\nThis is the problem I want to explore.\n\nThis isn't simply speculation about future AI systems.\n\nModern coding agents already depend on repository-level context.\n\nOpenAI's documentation for Codex recommends using persistent repository instructions such as `AGENTS.md`\n\nfor naming conventions, business logic, known quirks, dependencies, and other information that may not be inferable directly from code. It also recommends providing file paths, component names, diffs, and documentation when describing tasks.\n\nOpenAI has also described a broader approach where repository knowledge becomes a structured source of truth rather than one giant instruction document, explicitly noting that **context management is one of the biggest challenges for agents working on large and complex tasks**.\n\nThat leads to an interesting conclusion:\n\nIf context materially affects agent performance, context should also become part of agent evaluation.\n\nSWE-bench was created to evaluate AI systems on real software-engineering issues from GitHub repositories.\n\nThe agent receives a repository and an issue, modifies the code, and is evaluated using tests. SWE-bench Verified was later created as a human-validated subset after OpenAI and the SWE-bench authors found problems with some benchmark tasks. 500 tasks were selected after professional developers screened the data.\n\nBut benchmark methodology itself is evolving.\n\nIn February 2026, OpenAI reported that SWE-bench Verified had become increasingly contaminated and recommended newer evaluations such as SWE-bench Pro.\n\nIn July 2026, OpenAI also reported that its audit of SWE-bench Pro found widespread task-quality problems and estimated roughly 30% of tasks were broken.\n\nThat matters because it demonstrates a broader lesson:\n\nEvaluating AI systems is itself an engineering problem.\n\nA benchmark can produce a number without necessarily producing a reliable measurement.\n\nThis is also not an isolated idea.\n\nA 2026 research benchmark called **SWE-ContextBench** specifically investigates whether coding agents can reuse relevant experience across related software-engineering tasks.\n\nThe benchmark augments SWE-bench Lite with related tasks derived from dependency and reference relationships between GitHub issues and pull requests. It evaluates prediction accuracy, time efficiency, and cost efficiency. The authors report that appropriately selected summarized experience can improve resolution accuracy while reducing runtime and token cost, whereas poorly selected experience can provide limited or negative benefits.\n\nThat suggests something important:\n\n```\nMore context\n      ≠\nBetter result\n```\n\nThe real question is:\n\n```\nRelevant context\n        +\nCorrect retrieval\n        +\nCorrect interpretation\n        ↓\nBetter decision\n```\n\nI propose thinking about agent evaluation as a multi-dimensional problem.\n\nInstead of:\n\n```\nTask\n ↓\nAgent\n ↓\nCode\n ↓\nTests\n ↓\nPass / Fail\n```\n\nwe could evaluate:\n\n```\n                 ┌───────────────┐\n                 │     TASK      │\n                 └───────┬───────┘\n                         │\n             ┌───────────┼───────────┐\n             ▼           ▼           ▼\n         Repository   Constraints   History\n             │           │           │\n             └───────────┼───────────┘\n                         ▼\n                  AI CODING AGENT\n                         │\n                         ▼\n                      DECISION\n                         │\n          ┌──────────────┼──────────────┐\n          ▼              ▼              ▼\n     Code Quality   Constraints    Context Fit\n          │              │              │\n          └──────────────┼──────────────┘\n                         ▼\n                    Final Score\n```\n\nThis is the idea I find particularly interesting.\n\nKeep the:\n\n```\nModel\nTask\nRepository\n```\n\nas constant as possible.\n\nThen change **one meaningful part of the context**.\n\nFor example:\n\n```\nDatabase:\nPostgreSQL\n\nArchitecture:\nRepository pattern\n\nConstraint:\nAll database access must go through repositories.\n```\n\nThe agent produces:\n\n```\nController\n   ↓\nService\n   ↓\nRepository\n   ↓\nPostgreSQL\n```\n\nGood.\n\nChange one relevant constraint:\n\n```\nDatabase:\nPostgreSQL\n\nArchitecture:\nEvent-driven\n\nConstraint:\nServices must communicate through events.\n```\n\nNow the appropriate implementation should change.\n\nIf the agent continues producing the old architecture, we can measure a **context adaptation failure**.\n\nWe shouldn't reward an agent merely for changing its answer.\n\nSuppose we change something irrelevant:\n\n```\nREADME formatting\n```\n\nThe architecture hasn't changed.\n\nThe agent should ideally make the same engineering decision.\n\nSo:\n\n```\nRelevant context changes\n        ↓\nDecision SHOULD change\n```\n\nwhile:\n\n```\nIrrelevant context changes\n        ↓\nDecision SHOULD remain stable\n```\n\nThis gives us two useful properties.\n\nDoes the agent react when relevant context changes?\n\nDoes the agent remain stable when irrelevant context changes?\n\nA practical benchmark could use paired or grouped scenarios:\n\n```\nTask T\nContext C1\n      ↓\n   Agent\n      ↓\nDecision D1\n\nTask T\nContext C2\n      ↓\n   Agent\n      ↓\nDecision D2\n```\n\nWhere:\n\n```\nC1 → C2\n```\n\ncontains a controlled change.\n\nThen evaluate:\n\n```\nWas the change relevant?\n        ↓\nShould the decision change?\n        ↓\nDid the agent change?\n        ↓\nWas the new decision correct?\n```\n\nAnd separately:\n\n```\nWas the context change irrelevant?\n        ↓\nShould the decision remain stable?\n        ↓\nDid the agent unnecessarily change?\n```\n\nI wouldn't claim these are established industry-standard metrics. They are a proposed framework that would need experimental validation.\n\nDid the implementation satisfy the task?\n\nDid the implementation respect explicit constraints?\n\nWhen relevant context changed, how often did the agent make the appropriate change?\n\nWhen irrelevant context changed, how often did the agent preserve the appropriate decision?\n\nDoes the change follow the project's established architecture and conventions?\n\nDid the change break previously working behavior?\n\nHow much context did the agent need to retrieve to make the correct decision?\n\nThis could eventually produce something like:\n\n```\nAgent Reliability Score\n│\n├── Functional Correctness\n├── Constraint Adherence\n├── Context Adaptation\n├── Context Stability\n├── Repository Consistency\n├── Regression Resistance\n└── Context Efficiency\n```\n\nA common reaction might be:\n\n\"Just give the model the entire repository.\"\n\nBut that's not necessarily a solution.\n\nOpenAI's own engineering discussion around Codex describes the problem with extremely large instruction documents: context is limited, important information can be crowded out, stale instructions can accumulate, and humans may stop maintaining them. Their approach is instead to use a concise map pointing toward deeper sources of truth.\n\nSo the problem isn't simply:\n\n```\nHow much context?\n```\n\nIt is:\n\n```\nWhich context?\nWhen?\nFrom where?\nHow current?\nHow reliable?\nHow relevant?\n```\n\nThat is a much more interesting systems problem.\n\nI think repository context should be treated as something that changes over time:\n\n```\nInitial Decision\n      ↓\nImplementation\n      ↓\nNew Requirement\n      ↓\nArchitecture Change\n      ↓\nDependency Change\n      ↓\nSecurity Change\n      ↓\nNew Decision\n```\n\nAn agent working on a long-lived repository therefore needs something closer to:\n\n```\nCurrent State\n+\nHistorical Decisions\n+\nActive Constraints\n+\nRepository Structure\n+\nRelevant Documentation\n```\n\nrather than simply:\n\n```\nPrompt + Code\n```\n\nTwo implementations can be functionally equivalent while only one fits the project.\n\nFor example:\n\n```\n# Implementation A\ncache_result()\n```\n\nversus:\n\n```\n# Implementation B\nawait cache_result()\n```\n\nBoth might pass a narrow test.\n\nBut the correct choice could depend on:\n\nThe code itself doesn't always contain the complete explanation.\n\nSometimes the most important information is **why the code was designed that way**.\n\nTraditional benchmark thinking often looks like:\n\n```\nFixed Task\n   ↓\nFixed Dataset\n   ↓\nFixed Evaluation\n   ↓\nScore\n```\n\nBut real repositories look more like:\n\n```\nTask\n ↓\nRepository evolves\n ↓\nRequirements change\n ↓\nDependencies change\n ↓\nArchitecture changes\n ↓\nSecurity constraints change\n ↓\nAgent receives new task\n```\n\nTherefore, a future benchmark could intentionally introduce controlled environmental changes.\n\nFor example:\n\n```\nVersion 1\n   ↓\nAgent decision\n\nVersion 2\n   ↓\nArchitecture changed\n\nVersion 3\n   ↓\nSecurity policy changed\n\nVersion 4\n   ↓\nDependency changed\n```\n\nThen measure whether the agent adapts correctly.\n\nOne simple way to visualize the experiment:\n\n| Decision Should Stay Same | Decision Should Change | |\n|---|---|---|\nAgent stays same |\n✅ Stable | ❌ Adaptation failure |\nAgent changes |\n❌ Instability | ✅ Adaptation success |\n\nThis is interesting because it separates two failure modes that ordinary pass/fail evaluation can hide.\n\nThe question isn't:\n\n\"Can AI write code?\"\n\nWe're already measuring that.\n\nThe more difficult question is:\n\n\"Can an AI agent maintain correct engineering judgment as the software environment changes?\"\n\nThat includes:\n\n```\nArchitecture\nRequirements\nDependencies\nSecurity\nPerformance\nBusiness Rules\nRepository History\nTeam Conventions\n```\n\nThis is closer to how real software development works.\n\nWhen investigating a complex AI engineering problem, I don't think one reasoning style is enough.\n\nSometimes we need a diagram.\n\nSometimes a benchmark.\n\nSometimes a root-cause analysis.\n\nSometimes a comparison.\n\nSometimes a threat model.\n\nSometimes a timeline.\n\nSometimes a first-principles explanation.\n\nSo I compiled a reusable set of **100 visual-thinking, explanation, analysis, and strategy lenses**.\n\nThese aren't claims about AI capability. They are **ways to structure thinking and communicate technical problems**.\n\n| # | Shortcut | Lens |\n|---|---|---|\n| 1 | `/handwritten` |\nNotebook-style handwritten notes |\n| 2 | `/visualize` |\nTurn ideas into visual explanations |\n| 3 | `/stickynotes` |\nOne idea per sticky note |\n| 4 | `/infographic` |\nInfographic layout |\n| 5 | `/diagram` |\nDraw a concept diagram |\n| 6 | `/flowchart` |\nStep-by-step flowchart |\n| 7 | `/mindmap` |\nCreate a mind map |\n| 8 | `/xray` |\nShow internal structure |\n| 9 | `/blueprint` |\nTechnical blueprint |\n| 10 | `/explodedview` |\nBreak object into components |\n| 11 | `/thenvsnow` |\nCompare past vs present |\n| 12 | `/timeline` |\nChronological timeline |\n| 13 | `/beforeafter` |\nTransformation comparison |\n| 14 | `/cutaway` |\nCutaway illustration |\n| 15 | `/anatomy` |\nExplain all parts |\n| 16 | `/layers` |\nLayer-by-layer architecture |\n| 17 | `/ecosystem` |\nShow all connected players |\n| 18 | `/journey` |\nShow end-to-end journey |\n| 19 | `/process` |\nExplain a complete process |\n| 20 | `/cycle` |\nVisualize recurring cycles |\n| 21 | `/roadmap` |\nLearning or execution roadmap |\n| 22 | `/dashboard` |\nDashboard with KPIs |\n| 23 | `/comparison` |\nSide-by-side comparison |\n| 24 | `/versus` |\nHead-to-head comparison |\n| 25 | `/scale` |\nCompare sizes visually |\n| 26 | `/evolution` |\nShow evolution over time |\n| 27 | `/future` |\nImagine future scenarios |\n| 28 | `/inside` |\nReveal inner workings |\n| 29 | `/microscopic` |\nZoom into microscopic detail |\n| 30 | `/macroscopic` |\nZoom out to system level |\n| 31 | `/crosssection` |\nCross-sectional illustration |\n| 32 | `/map` |\nGeographic or conceptual map |\n| 33 | `/heatmap` |\nShow intensity |\n| 34 | `/network` |\nShow relationships |\n| 35 | `/architecture` |\nSoftware/system architecture |\n| 36 | `/wireframe` |\nWebsite/app layout |\n| 37 | `/mockup` |\nRealistic product preview |\n| 38 | `/prototype` |\nEarly product concept |\n| 39 | `/schematic` |\nSimple technical schematic |\n| 40 | `/isometric` |\n3D isometric illustration |\n| 41 | `/birdseye` |\nTop-down view |\n| 42 | `/360view` |\nAll-angle visualization |\n| 43 | `/storyboard` |\nScene-by-scene explanation |\n| 44 | `/comic` |\nExplain through comic panels |\n| 45 | `/poster` |\nPoster design |\n| 46 | `/cover` |\nBook/report cover |\n| 47 | `/adcreative` |\nAdvertising concept |\n| 48 | `/thumbnail` |\nYouTube thumbnail concept |\n| 49 | `/carousel` |\nInstagram/LinkedIn carousel |\n| 50 | `/socialvisual` |\nSocial media graphic |\n| 51 | `/quotevisual` |\nQuote as shareable visual |\n| 52 | `/eli5` |\nExplain simply |\n| 53 | `/expert` |\nExpert-level explanation |\n| 54 | `/firstprinciples` |\nBreak down to fundamentals |\n| 55 | `/deepdive` |\nComprehensive explanation |\n| 56 | `/simplify` |\nSimplify difficult content |\n| 57 | `/analogy` |\nExplain through analogy |\n| 58 | `/socratic` |\nTeach through questions |\n| 59 | `/teachme` |\nStructured tutoring |\n| 60 | `/cheatsheet` |\nQuick-reference notes |\n| 61 | `/flashcards` |\nStudy flashcards |\n| 62 | `/quiz` |\nGenerate a quiz |\n| 63 | `/viva` |\nViva preparation |\n| 64 | `/interview` |\nMock interview |\n| 65 | `/devilsadvocate` |\nChallenge assumptions |\n| 66 | `/factcheck` |\nVerify claims |\n| 67 | `/mythvsfact` |\nSeparate myths from facts |\n| 68 | `/proscons` |\nAdvantages vs disadvantages |\n| 69 | `/swot` |\nSWOT analysis |\n| 70 | `/pestle` |\nPESTLE analysis |\n| 71 | `/fiveforces` |\nPorter's Five Forces |\n| 72 | `/rootcause` |\nFind root cause |\n| 73 | `/fivewhys` |\nFive Whys analysis |\n| 74 | `/decisionmatrix` |\nWeighted decision matrix |\n| 75 | `/scenario` |\nScenario planning |\n| 76 | `/simulate` |\nSimulation exercise |\n| 77 | `/roleplay` |\nAssume an expert role |\n| 78 | `/consultant` |\nConsulting-style advice |\n| 79 | `/executivebrief` |\nExecutive summary |\n| 80 | `/insights` |\nExtract insights |\n| 81 | `/recommendations` |\nProvide recommendations |\n| 82 | `/prioritize` |\nRank by priority |\n| 83 | `/benchmark` |\nBenchmark comparison |\n| 84 | `/marketmap` |\nIndustry landscape |\n| 85 | `/strategy` |\nStrategic planning |\n| 86 | `/businessmodel` |\nBusiness model explanation |\n| 87 | `/pitch` |\nInvestor/startup pitch |\n| 88 | `/investor` |\nInvestor perspective |\n| 89 | `/redteam` |\nStress-test a plan |\n| 90 | `/premortem` |\nAssume failure and analyze why |\n| 91 | `/reverseengineer` |\nBreak down success |\n| 92 | `/promptengineer` |\nOptimize prompts |\n| 93 | `/research` |\nStructured research |\n| 94 | `/sources` |\nFind reliable sources |\n| 95 | `/summarize` |\nSummarize content |\n| 96 | `/extract` |\nExtract key information |\n| 97 | `/table` |\nConvert into a table |\n| 98 | `/presentation` |\nPresentation outline |\n| 99 | `/dashboardanalysis` |\nAnalyze dashboards |\n| 100 | `/actionplan` |\nCreate step-by-step action plan |\n\nThese 100 lenses are not 100 claims that an AI model is more intelligent when using them.\n\nThey are simply **structured ways of looking at a problem**.\n\nFor AI-agent research, different lenses can answer different questions:\n\n```\n/architecture\n        ↓\nWhat is the system structure?\n\n/xray\n        ↓\nWhat is happening internally?\n\n/timeline\n        ↓\nHow did the system change?\n\n/thenvsnow\n        ↓\nWhat changed between versions?\n\n/benchmark\n        ↓\nHow should we measure it?\n\n/factcheck\n        ↓\nWhich claims have evidence?\n\n/redteam\n        ↓\nHow can the evaluation fail?\n\n/rootcause\n        ↓\nWhy did the agent fail?\n\n/decisionmatrix\n        ↓\nWhich approach is better?\n\n/actionplan\n        ↓\nWhat should we build next?\n```\n\nThis is especially useful when researching complex AI systems because no single representation captures the entire problem.\n\nIf I were turning this idea into an actual research experiment, I'd start small.\n\nCreate 50–100 repository-level tasks.\n\nFor each task, create controlled variants:\n\n```\nArchitecture change\nRequirement change\nSecurity constraint change\nDependency change\nPerformance constraint change\nDocumentation change\nIrrelevant formatting change\n```\n\nRun:\n\n```\nSame Model\nSame Task\nDifferent Context\n```\n\nThen measure:\n\n```\nFunctional correctness\nContext adaptation\nContext stability\nConstraint adherence\nRegression\nToken usage\nRuntime\n```\n\nFor ambiguous cases, use experienced developers to verify whether the changed decision was actually appropriate.\n\nThis is important because benchmark design itself can introduce errors. OpenAI's SWE-bench work demonstrates why human validation and benchmark auditing matter when interpreting agent performance.\n\nMy current hypothesis is:\n\nA reliable coding agent should not simply produce correct code. It should produce decisions that are appropriate for the current context, adapt when relevant context changes, and remain stable when irrelevant context changes.\n\nThat's a much stronger definition of reliability.\n\nAnd importantly, it is something we can attempt to measure.\n\nAI coding agents are moving from:\n\n```\nCode Completion\n```\n\ntoward:\n\n```\nSoftware Engineering Agents\n```\n\nAs that transition happens, our evaluation methods need to evolve too.\n\nThe future benchmark may not simply ask:\n\n“Did the code pass?”\n\nIt may need to ask:\n\n“Did the agent understand the current system well enough to make the right engineering decision?”\n\nThat is the problem I find most interesting.\n\nAnd I don't think we have completely solved it yet.\n\n**What would you add to a context-aware coding-agent benchmark?**\n\nArchitecture changes?\n\nSecurity constraints?\n\nDependency changes?\n\nBusiness requirements?\n\nRepository history?\n\nI'd genuinely like to hear how other developers would design it.\n\n`AGENTS.md`\n\n, task specification, and development-environment guidance.", "url": "https://wpnews.pro/news/beyond-passing-tests-a-100-lens-framework-for-evaluating-context-aware-ai-coding", "canonical_source": "https://dev.to/probal_dhali_f7d15eac866a/beyond-passing-tests-a-100-lens-framework-for-evaluating-context-aware-ai-coding-agents-om", "published_at": "2026-08-24 15:42:58+00:00", "updated_at": "2026-08-24 16:14:01.340610+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-tools", "developer-tools"], "entities": ["OpenAI", "Codex", "SWE-bench", "SWE-bench Pro", "SWE-ContextBench", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/beyond-passing-tests-a-100-lens-framework-for-evaluating-context-aware-ai-coding", "markdown": "https://wpnews.pro/news/beyond-passing-tests-a-100-lens-framework-for-evaluating-context-aware-ai-coding.md", "text": "https://wpnews.pro/news/beyond-passing-tests-a-100-lens-framework-for-evaluating-context-aware-ai-coding.txt", "jsonld": "https://wpnews.pro/news/beyond-passing-tests-a-100-lens-framework-for-evaluating-context-aware-ai-coding.jsonld"}}