AgentRuleBench tests a widely repeated claim: that AI coding agents drift from the architectural conventions you write in prose (CLAUDE.md, AGENTS.md, GEMINI.md), so you need a deterministic lint rule to hold the boundary.
On the rule measured here, they did not drift. In a pilot across three vendors' agents and all four conditions (from an unguarded control to run-lint-and-fix), and in a pre-registered exploration that added a deliberately baited task and cheaper models, current agents did not import a UI component into a request-entry file. This repository is the harness, the pre-registration, and the raw run data, so you can rerun it and check the result yourself.
Read the full write-up in WRITEUP.md: the premise, the method, the numbers, and a careful account of what the null does and does not mean.
The rule under test was inferred by archprint, an open-source tool (on npm) that mines architecture rules from a repository's real import graph and gates each on statistical evidence. AgentRuleBench and archprint are released together; this repository is the benchmark, archprint is the tool.
WRITEUP.md: the full write-up, and the basis for the public post.pre-registration/: the design, written before the runs (PREREGISTRATION.md and the follow-up exploration).runner/: the provider-agnostic agent loop and the four experimental conditions.score/: the independent TypeScript-AST compliance scorer (it does not reuse the lint rule it checks).tasks/: the two coding tasks, a single-route and a three-route feature.results/: the actual pilot and exploration run data (JSONL).corpus/: the near-census corpus validation and composition that back the scale numbers in the write-up.docs/REPRODUCE.md: exact setup and run steps, plus the model and cost table.
npm install
cp env/.env.example env/.env # add the provider keys you want to run
bash scripts/fetch-repos.sh # clone the benchmark repo at its pinned SHA
npm run check-env # confirm which keys are visible
npm run pilot # or: npm run gate
Full instructions, the exact models, trial counts, and estimated cost are in docs/REPRODUCE.md.
This is a narrow, specific finding, not "AI agents respect architecture." Only one boundary was tested at the agent level (a request entry must not import a UI component), on one repository. A second inferred boundary (no direct database import in a request entry) was analyzed at corpus scale and dropped as a benchmark rule, because roughly a quarter to a third of real apps do it, so it is idiomatic rather than a clean violation. Everything beyond the TypeScript import boundary is out of scope. See WRITEUP.md for the full account.
MIT. Copyright (c) 2026 Oluwatomiwa Ajiferuke. See LICENSE.