{"slug": "show-hn-agentrulebench-does-ai-agents-violate-inferred-architecture-rules", "title": "Show HN: AgentRuleBench, does AI agents violate inferred architecture rules?", "summary": "AgentRuleBench, an open-source benchmark released alongside the archprint architecture-rule mining tool, found that AI coding agents did not violate an inferred architecture rule across a pilot spanning three vendors' agents and four experimental conditions, plus a pre-registered exploration that added a deliberately baited task and cheaper models. The rule tested was that a request-entry file must not import a UI component; the benchmark's author, Oluwatomiwa Ajiferuke, states the finding is narrow and does not support the broader claim that \"AI agents respect architecture.\" A second inferred boundary barring direct database imports in request entries was dropped as a benchmark rule because roughly a quarter to a third of real apps do it, making it idiomatic rather than a clean violation.", "body_md": "AgentRuleBench tests a widely repeated claim: that AI coding agents drift from the architectural conventions you write in prose (CLAUDE.md, AGENTS.md, GEMINI.md), so you need a deterministic lint rule to hold the boundary.\n\nOn the rule measured here, they did not drift. In a pilot across three vendors' agents and all four conditions (from an unguarded control to run-lint-and-fix), and in a pre-registered exploration that added a deliberately baited task and cheaper models, current agents did not import a UI component into a request-entry file. This repository is the harness, the pre-registration, and the raw run data, so you can rerun it and check the result yourself.\n\nRead the full write-up in [WRITEUP.md](/Tommkruix/agentrulebench/blob/main/WRITEUP.md): the premise, the method, the numbers, and a careful account of what the null does and does not mean.\n\nThe rule under test was inferred by [archprint](https://github.com/Tommkruix/archprint), an open-source tool (on npm) that mines architecture rules from a repository's real import graph and gates each on statistical evidence. AgentRuleBench and archprint are released together; this repository is the benchmark, archprint is the tool.\n\n- `WRITEUP.md` : the full write-up, and the basis for the public post.\n- `pre-registration/` : the design, written before the runs (PREREGISTRATION.md and the follow-up exploration).\n- `runner/` : the provider-agnostic agent loop and the four experimental conditions.\n- `score/` : the independent TypeScript-AST compliance scorer (it does not reuse the lint rule it checks).\n- `tasks/` : the two coding tasks, a single-route and a three-route feature.\n- `results/` : the actual pilot and exploration run data (JSONL).\n- `corpus/` : the near-census corpus validation and composition that back the scale numbers in the write-up.\n- `docs/REPRODUCE.md` : exact setup and run steps, plus the model and cost table.\n\n```\nnpm install\ncp env/.env.example env/.env      # add the provider keys you want to run\nbash scripts/fetch-repos.sh       # clone the benchmark repo at its pinned SHA\nnpm run check-env                 # confirm which keys are visible\nnpm run pilot                     # or: npm run gate\n```\n\nFull instructions, the exact models, trial counts, and estimated cost are in [docs/REPRODUCE.md](/Tommkruix/agentrulebench/blob/main/docs/REPRODUCE.md).\n\nThis is a narrow, specific finding, not \"AI agents respect architecture.\" Only one boundary was tested at the agent level (a request entry must not import a UI component), on one repository. A second inferred boundary (no direct database import in a request entry) was analyzed at corpus scale and dropped as a benchmark rule, because roughly a quarter to a third of real apps do it, so it is idiomatic rather than a clean violation. Everything beyond the TypeScript import boundary is out of scope. See [WRITEUP.md](/Tommkruix/agentrulebench/blob/main/WRITEUP.md) for the full account.\n\nMIT. Copyright (c) 2026 Oluwatomiwa Ajiferuke. See [LICENSE](/Tommkruix/agentrulebench/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-agentrulebench-does-ai-agents-violate-inferred-architecture-rules", "canonical_source": "https://github.com/Tommkruix/agentrulebench", "published_at": "2026-09-12 19:57:08+00:00", "updated_at": "2026-09-12 20:25:28.383039+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-research"], "entities": ["AgentRuleBench", "archprint", "Oluwatomiwa Ajiferuke", "npm", "TypeScript"], "alternates": {"html": "https://wpnews.pro/news/show-hn-agentrulebench-does-ai-agents-violate-inferred-architecture-rules", "markdown": "https://wpnews.pro/news/show-hn-agentrulebench-does-ai-agents-violate-inferred-architecture-rules.md", "text": "https://wpnews.pro/news/show-hn-agentrulebench-does-ai-agents-violate-inferred-architecture-rules.txt", "jsonld": "https://wpnews.pro/news/show-hn-agentrulebench-does-ai-agents-violate-inferred-architecture-rules.jsonld"}}