Show HN: AgentRuleBench, does AI agents violate inferred architecture rules?

AgentRuleBench, an open-source benchmark released alongside the archprint architecture-rule mining tool, found that AI coding agents did not violate an inferred architecture rule across a pilot spanning three vendors' agents and four experimental conditions, plus a pre-registered exploration that added a deliberately baited task and cheaper models. The rule tested was that a request-entry file must not import a UI component; the benchmark's author, Oluwatomiwa Ajiferuke, states the finding is narrow and does not support the broader claim that "AI agents respect architecture." A second inferred boundary barring direct database imports in request entries was dropped as a benchmark rule because roughly a quarter to a third of real apps do it, making it idiomatic rather than a clean violation.

AgentRuleBench tests a widely repeated claim: that AI coding agents drift from the architectural conventions you write in prose CLAUDE.md, AGENTS.md, GEMINI.md , so you need a deterministic lint rule to hold the boundary. On the rule measured here, they did not drift. In a pilot across three vendors' agents and all four conditions from an unguarded control to run-lint-and-fix , and in a pre-registered exploration that added a deliberately baited task and cheaper models, current agents did not import a UI component into a request-entry file. This repository is the harness, the pre-registration, and the raw run data, so you can rerun it and check the result yourself. Read the full write-up in WRITEUP.md /Tommkruix/agentrulebench/blob/main/WRITEUP.md : the premise, the method, the numbers, and a careful account of what the null does and does not mean. The rule under test was inferred by archprint https://github.com/Tommkruix/archprint , an open-source tool on npm that mines architecture rules from a repository's real import graph and gates each on statistical evidence. AgentRuleBench and archprint are released together; this repository is the benchmark, archprint is the tool. - WRITEUP.md : the full write-up, and the basis for the public post. - pre-registration/ : the design, written before the runs PREREGISTRATION.md and the follow-up exploration . - runner/ : the provider-agnostic agent loop and the four experimental conditions. - score/ : the independent TypeScript-AST compliance scorer it does not reuse the lint rule it checks . - tasks/ : the two coding tasks, a single-route and a three-route feature. - results/ : the actual pilot and exploration run data JSONL . - corpus/ : the near-census corpus validation and composition that back the scale numbers in the write-up. - docs/REPRODUCE.md : exact setup and run steps, plus the model and cost table. npm install cp env/.env.example env/.env add the provider keys you want to run bash scripts/fetch-repos.sh clone the benchmark repo at its pinned SHA npm run check-env confirm which keys are visible npm run pilot or: npm run gate Full instructions, the exact models, trial counts, and estimated cost are in docs/REPRODUCE.md /Tommkruix/agentrulebench/blob/main/docs/REPRODUCE.md . This is a narrow, specific finding, not "AI agents respect architecture." Only one boundary was tested at the agent level a request entry must not import a UI component , on one repository. A second inferred boundary no direct database import in a request entry was analyzed at corpus scale and dropped as a benchmark rule, because roughly a quarter to a third of real apps do it, so it is idiomatic rather than a clean violation. Everything beyond the TypeScript import boundary is out of scope. See WRITEUP.md /Tommkruix/agentrulebench/blob/main/WRITEUP.md for the full account. MIT. Copyright c 2026 Oluwatomiwa Ajiferuke. See LICENSE /Tommkruix/agentrulebench/blob/main/LICENSE .