# How we tested ArchDocs against agent rules files

> Source: <https://archdocs.io/blog/how-we-tested-archdocs-against-agent-rules-files>
> Published: 2026-11-06 00:00:00+00:00

# How we tested ArchDocs against agent rules files

How we tested coding agents using CLAUDE.md and AGENTS.md against ArchDocs task context: lesson census, blind grading, run accounting and post-hoc changes.

- Author
- The ArchDocs team
- Published
- Reading
- 25 min read

This is the method behind [I tried to prove ArchDocs beats a good CLAUDE.md](https://archdocs.io/blog/i-tried-to-prove-archdocs-beats-a-good-claude-md). The question was whether a coding agent using our real rules files misses lessons our team has already recorded, and whether ArchDocs task context changes what the agent builds. Task context is the ArchDocs call (the `get_task_context` MCP tool) that takes a task description, a repository and file paths, and returns the matching design docs, decisions, findings and Work Log entries.

Everything ran on our own records over November 5 and 6, 2026, with this study's own records filtered out of what any agent could see. There were four stages: a pilot, a census of lessons, a test of what reaches the agent at task start, and a coding trial. The retrieval stage doubled as development work, so its numbers are development-set results. The coding trial used five valid tasks selected to be able to show a difference. Neither stage is a sample of ordinary engineering work.

## [Terms](#terms)

- Baseline: the repository at a fixed commit, its git history, and the rules files we actually use (CLAUDE.md, or AGENTS.md copied from it).
- ArchDocs: the baseline plus ArchDocs task context.
- Explicit lesson: the baseline with the lesson's one- or two-sentence statement appended to the root AGENTS.md. This setup exists only in the coding trial.
- Task: one brief given to an agent. Run: one attempt at a task in one setup. Completed run: a run that produced a patch we could grade.
- Conveyed lesson: delivered text that states the whole rule. Pointer: delivered text that names the record holding the rule without stating it.

## [What was fixed in advance and what changed later](#what-was-fixed-in-advance-and-what-changed-later)

We wrote five ground rules before collecting results:

1. The baseline never loses because information it would reasonably have was withheld.
2. Candidate lessons come from real records by a rule fixed before outcomes are seen, and every candidate is counted.
3. Tasks the code alone solves are screened out.
4. Every batch includes controls where the lesson is already in a rules file.
5. Graders are checked on known implementations before any run, and any change after outputs are seen is logged.

Rules 1 and 4 were not met in the coding trial, as described below. Everything else that changed falls into one of these groups.

| Kind of change | What changed | When | 
|---|---|---|
| Product fixes | Task-context matching, decision and Work Log indexing, a capped meaning-based matcher | During the pilot and retrieval stages, tuned on the evaluation cases | 
| Design changes | Codex instead of Claude Code for the trial; explicit-lesson setup instead of natural controls; a fifth task from set B | After screening, before any trial run | 
| Setup fixes | A Claude-only instruction removed from every AGENTS.md; patch capture narrowed | After the first trial start | 
| Grader repairs | Entry-point discovery; required request fields | After seeing outputs, regraded blind to setup | 
| Analysis choices | Fisher's exact test; a "readable tasks" filter, later dropped; an exclusion of the all-pass task | After seeing results | 

## [What information each setup had](#what-information-each-setup-had)

The baseline could always read any file in its repository and search its git history. A lesson written in a spec, plan or commit message was available to it, but only its rules files were delivered when a task started. ArchDocs adds a retrieval step over the records we kept in our workspace.

We planned one more correction for the baseline. Some lessons exist only in ArchDocs because this team now writes specs and plans there. A team without ArchDocs would probably keep those as repository files, so the plan said the baseline would get a repository copy. This was applied in the pilot only.

| Stage | Baseline received | ArchDocs received | Repository copies of ArchDocs-only records | 
|---|---|---|---|
| Pilot | Lesson sources as path-scoped rules files | Lesson sources as imported documents | Yes: both setups got the same sources | 
| Delivery test | Rules files that load for the task's paths | One task-context response | Not applicable: the test measured delivered text only | 
| Coding trial | Repository, history and AGENTS.md | Baseline plus the ArchDocs plugin | No | 

In the coding trial, the census lists three of the five valid tasks' lessons as existing only in ArchDocs: team archive and both review-upload tasks. For those three, the baseline could not have found the rule by searching. Both review-upload lessons come from a design review for a feature whose code wasn't merged yet. The results show what happens when a decision lives only in ArchDocs. They don't measure ArchDocs against a repository that keeps the same plan as a file.

## [Stage 1: a ten-run pilot that tied](#stage-1-a-ten-run-pilot-that-tied)

Five tasks, each built around a lesson from our history, ran once per setup. The agent was Claude Code with Sonnet in a container with no network, on our monorepo at commit 320a0149. We removed the written lesson statements and their source files from that tree, then gave the sources back to each setup in its own form. The ArchDocs side used a separate workspace holding the nine imported source documents.

All ten runs passed both the feature test and the lesson test. The code next to each task already encoded the lesson, and every patch reused the existing helper. The agents read files through shell commands, so path-scoped rules, which load when an agent opens a matching file, never loaded for the baseline. Task context returned nothing in four of five ArchDocs runs, because it matched the whole task sentence or exact two-word phrases from it. We fixed that matching before the next stage. The ArchDocs runs used more total input tokens in all five pairs. Their first request carried about 14,000 more tokens of tool definitions and instructions.

## [Stage 2: counting real lessons](#stage-2-counting-real-lessons)

We built a census of lessons from two sources. The repository side covered our monorepo (at c81b8ca3) and our public plugin repository (at 693a014b): every CLAUDE.md rule, 379 fix commits from the previous 120 days, normative statements in specs and plans, the plugin's skills and couplings between the two repositories. It produced 187 lessons. The organization side covered 53 design docs in our workspace, including their Work Log findings, evidence, knowledge items, answered questions and relations. Four sample fixtures among those 53 contributed nothing. It produced 203 lessons, 71 of them duplicates of repository lessons, for 319 in total.

Codex models extracted the lessons read-only. Automated checks confirmed that every lesson had its required fields and that every cited source and line existed at the pinned commits. Beyond that, the extraction was spot-checked only informally: a few lessons were read against their sources, and one of those turned up a stale Free-plan limit in our plugin's instructions. We didn't draw a random sample or measure an error rate, so every count below carries the extraction model's judgement. A lesson the model missed doesn't appear at all.

The census answers three separate questions, and the rows of each block add up to 319.

| Question | Answer | Lessons | Share | 
|---|---|---|---|
| Is it in a rules file today? | Yes | 103 | 32% | 
|  | No rules file found | 216 | 68% | 
| Where is it written? | Somewhere in the examined repositories | 292 | 92% | 
|  | Only in ArchDocs (confirmed) | 18 | 6% | 
|  | Unknown: applies to a repository we didn't examine | 9 | 3% | 
| Does nearby code show it? | Yes | 187 | 59% | 
|  | Partly | 47 | 15% | 
|  | No | 79 | 25% | 
|  | Unknown: same unexamined repository | 6 | 2% | 

"Only in ArchDocs" means no equivalent statement was found in either examined repository or its reachable commit messages, and nearby code doesn't encode it. The nine unknowns apply to our benchmark repository, which wasn't supplied to the census. If none of them appears there, the ArchDocs-only share rises from 18 of 319 (5.6%) to at most 27 of 319 (8.5%). If some appear there, the rules-file share could rise from 103 to as many as 112.

Two conclusions follow for our team. ArchDocs rarely holds information the repositories lack outright. Most lessons never reach a file an agent loads automatically, so the useful question is whether a lesson reaches the agent when a task starts.

## [Stage 3: what reaches the agent at task start](#stage-3-what-reaches-the-agent-at-task-start)

### [Cases](#cases)

We took every organization-census lesson whose code inferability was "no" (59) or "partly" (26): 85 cases. For each we wrote a neutral task brief with the target repository (61 monorepo, 24 plugin) and one to five real file paths, and checked the brief for leaked answers. 24 cases were controls, lessons stated in a rules file that loads for those paths. The other 61 had no loaded rules file stating them.

Other repository text, meaning any tracked file that doesn't load automatically, stated the full rule for 13 of the 61, part of it for 19 and none of it for 29. Those 32 partial or full statements were available to a baseline agent that searched. None of them were delivered.

### [Delivery and judging](#delivery-and-judging)

For the baseline, the delivered text was the rules files that load at session start for those paths. For ArchDocs, it was one task-context response, run directly against our production API so the text was verbatim. A blind judge (Codex on gpt-6-astra, read-only) saw the lesson and the delivered text with the setup hidden, and graded each case as conveyed, pointer or neither. Judging identical baseline text in two separate runs, the judge changed its verdict on one case.

### [These are development-set results](#these-are-development-set-results)

We used these 85 cases to find defects and tune the fix:

- Answered design doc questions weren't indexed, and Work Log text could make a document match while the returned passage came from elsewhere. We fixed both and reindexed every live design doc in production.
- An offline probe on the same 61 cases compared word matching with a small local embedding model (all-MiniLM-L6-v2). Blending the two put the lesson's document in the top three 72% of the time, against 41% for words alone.
- The first production blend returned every visible record ranked by similarity, so the same unrelated records reached every response and pages grew from about 8 items to 13. We tried caps of 3, 5 and 10, judged 5 and 10 on the same cases, and shipped 10. It runs for our own workspace only, behind an allowlist.

No held-out set exists. The results below show how far the changes moved these 85 cases, not how task context performs on lessons it wasn't tuned on.

| ArchDocs build | 61 lessons not in rules files (conveyed / pointer) | 24 controls (conveyed / pointer) | 
|---|---|---|
| Word matching after the indexing fixes (04d3d584) | 8 / 22 | 1 / 5 | 
| Words plus meaning, uncapped (fa1bb2cc) | 6 / 22 | 1 / 4 | 
| Words plus meaning, capped at ten (7ebc8ae5) | 10 / 31 | 0 / 7 | 

Rules files conveyed all 24 controls by construction, and 1 of the 61. With rules files and the capped build together, 35 of 85 lessons were conveyed, against 25 for rules files alone.

For the paired comparison, each of the 61 cases has an outcome per build: hit (conveyed or pointer) or miss. Between the word-only build and the capped build, 11 misses became hits (2 conveyed, 9 pointers) and no hit became a miss. A two-sided exact McNemar test on those 11 discordant pairs gives p = 0.001. Because the cap was chosen on these cases and the judge varies by about a case between runs, it isn't validation. Most of the gain is in pointers, which still leave the agent to open the record and find the rule.

## [Stage 4: the coding trial](#stage-4-the-coding-trial)

### [Candidate flow](#candidate-flow)

Before screening, we split the 85 delivery cases into three sets and fixed their order with seed 20261105:

- Set A: the 10 lessons the capped build conveyed in full, all screened.
- Set B: the other 51 non-controls, in random order. The rule was to take the first 4 that pass screening.
- Set C: the 24 controls, in random order. The rule was to take the first 3 that pass.

Set A deliberately put lessons that retrieval already handled at the front of the queue. A fixed seed makes the order reproducible. It doesn't make the selected tasks representative.

Codex (gpt-6-astra, read-only) screened each candidate. It passed only if a small realistic task could be written where an agent without the lesson would likely go wrong, an agent with it would get it right, and an automated test could tell them apart.

| Round | Screened | Passed | 
|---|---|---|
| 1 | Set A: all 10. Set B: positions 1 to 10. Set C: positions 1 to 6 | 3: one from A (view refresh), two from B (release check, bulk team members) | 
| 2 | Set B: positions 11 to 30. Set C: positions 7 to 24 | 3, all from B: both review-upload tasks and team archive | 
| Not screened | Set B: positions 31 to 51 (21 cases) | Set B already had 4 passes | 

Round 2's range was fixed before its results. Set B's fourth pass came at position 17, and team archive was its fifth, at position 28. Of the 58 failures across all 64 screened candidates, 28 were gradable but nearby code already shows the pattern, and 30 couldn't be graded by a code test at all (release order, approvals, how an agent reports its work). Six of those 30 were also visible in code.

No control passed, which broke rule 4. We replaced natural controls with the explicit-lesson setup. Because set C supplied nothing, we also used the spare fifth set B task instead of stopping at four. Both changes were made after screening and before any trial run, and were approved as part of the 54-run budget. The headline team-archive task is that spare.

Six tasks went into the trial. One, adding several people to a team at once, turned out invalid: every run in every setup built it as frontend batching over the existing single-member endpoint, and the feature test needed a bulk HTTP operation the brief never asked for. That leaves five valid tasks. The view-refresh task passed screening, but every run in every setup then followed the lesson, so it was a screening miss. We kept it in the results because no exclusion rule for it was set in advance.

### [Setups](#setups)

Every run used Codex CLI 0.166.0 with gpt-6.1-sol at high reasoning, with a 45-minute limit, in a fresh container on an internal network. Its only route out was a proxy allowing chatgpt.com, auth.openai.com and api.openai.com, plus the ArchDocs filter for the ArchDocs setup. Each repository was at a fixed commit (monorepo 7ebc8ae5, plugin e0a394fb) with history up to that commit and no other refs. The run order was seeded (20261106) so that every block of three starts contained one run of each setup.

- Baseline: an AGENTS.md generated beside every CLAUDE.md, byte for byte except one removed section (see run accounting below).
- ArchDocs: the baseline plus the ArchDocs Codex plugin, installed with Codex's own plugin commands and its hooks trusted through the normal hash-scoped flow. We used the committed 0.19.0 build, which wasn't released yet: it is the released 0.18.0 plus one new skill for reviewing a design doc and uploading the review. The plugin's MCP address pointed at a filtering proxy that exposed 21 read-only tools, refused every write and removed this study's records at any depth.
- Explicit lesson: the baseline plus a `## Lessons` section at the end of the root AGENTS.md holding the task's rule as one or two sentences. For team archive it read: "The draft Teams repair requires checking approver-pool dependencies before archiving or deleting a team, so documents cannot lose all eligible approvers."

The explicit-lesson setup shows whether a task depends on the lesson. It isn't the best possible baseline. It gets a short statement, while an ArchDocs agent can read the underlying design doc. On the review-provenance task, the ArchDocs agents read the design review's answered questions in full, and the explicit-lesson setup got two sentences. That row doesn't show that retrieval beats supplying equivalent context.

### [Graders](#graders)

Each task had a feature test, which any reasonable implementation of the brief should pass, and a lesson test, which fails when the lesson is ignored. The graders drove behaviour from the outside: HTTP routes through the real Nest app on a private Postgres database, React hooks through a DOM, or the plugin's command line. Small discovery adapters handled naming differences. Codex wrote the graders and their reference implementations. Before any run, each grader had to show four things: an empty patch fails, a correct implementation passes, a plausible implementation that ignores the lesson passes the feature test and fails the lesson test, and a differently structured correct implementation passes. Each empty patch ran once and each of the other three ran twice, with identical results.

Two repairs happened after we had seen outputs. Both followed the same procedure. We copied the affected patches under random IDs with no setup labels. Codex changed only how the grader finds and calls the submission, never what it asserts. It revalidated against the references, then regraded every affected patch. Both grader diffs are kept.

| Repair | Noticed when | Patches regraded | Outcome changes | 
|---|---|---|---|
| Entry-point discovery (release check, bulk members) | Explicit-lesson runs failed because the grader couldn't find their entry points | 18 | None | 
| Required request fields (both upload tasks) | ArchDocs runs failed because the grader omitted a `model` field the design doc requires | 19: all 17 completed upload runs plus 2 renamed-migration variants | One ArchDocs review-provenance run now passed | 

The second repair was noticed while reading ArchDocs failures, and its only effect was to turn an ArchDocs failure into a pass. It was applied blind to setup and changed no assertion.

### [Run accounting](#run-accounting)

| Item | Runs | 
|---|---|
| Planned starts | 54 (6 tasks × 3 setups × 3) | 
| Lost before any code change | 1 (start 1, ArchDocs, review provenance) | 
| Completed and graded | 53 | 
| On the invalid task | 9 | 
| In the results table | 44 (baseline 15, ArchDocs 14, explicit lesson 15) | 

Start 1 was lost to our own setup. CLAUDE.md tells a Claude session to hand implementation to Codex through a skill. In the generated AGENTS.md, Codex read that as a hard requirement, couldn't find the skill and stopped without changing anything. We removed that section from every setup's AGENTS.md for the remaining starts. The lost start counts against the 54 and isn't replaced, so the ArchDocs review-provenance cell has two runs. It's an experiment setup failure, not a product outcome, so it's out of the denominators. We also excluded files the setup itself added from patch capture, keeping a full diff for audit.

Three runs hit the 45-minute limit and were graded on the patch they had written: two ArchDocs view-refresh runs, both of which passed, and one baseline run on the invalid task. Our batch runner was killed by a two-hour tool limit with three runs in flight. The containers kept running, and we collected those three patches afterwards, though their event logs are incomplete. They are graded like any other run.

We had no written retry rule. In practice, we regraded a grading attempt once when it failed before any assertion ran. That happened twice: an explicit-lesson view-refresh grade timed out under machine load and passed on regrade, and one blind bulk-members grade ran out of memory during Prisma generation. Migration failures were never retried. Two ArchDocs runs, one on each upload task, dated a migration so that it ran before a table it depends on existed, which fails on a fresh database. They count as failures, as they would in CI. With the migration renamed to sort last, the review-pricing run passed the feature test and failed all four lesson checks: it built no separate upload setting. The review-provenance run passed one of two lesson checks: it kept a verification timestamp supplied by the uploading client. Neither would have passed.

### [Results](#results)

Each cell counts runs that passed both the feature and the lesson test.

| Lesson | Baseline | ArchDocs | Explicit lesson | 
|---|---|---|---|
| Archiving a team must check whether documents depend on it for approvers | 0/3 | 3/3 | 3/3 | 
| Uploaded agent reviews keep who sent them, and unchecked citations stay labelled as reported | 0/3 | 1/2 | 0/3 | 
| Agent review uploads are free on every plan, behind their own setting | 0/3 | 0/3 | 3/3 | 
| A release check passes a scenario on two of three tries | 0/3 | 0/3 | 0/3 | 
| A save refreshes each affected view once and closes the editor without waiting for it | 3/3 | 3/3 | 3/3 | 

On the release-check task, explicit-lesson runs passed 8 of 9 lesson checks each, against 5 or 6 for the other setups. On the review-provenance task, two of the three explicit-lesson runs also failed the feature test.

At the task level, ArchDocs did better than the baseline on two tasks, the same on three and worse on none. Over the five, ArchDocs runs passed 7 of 14, baseline runs 3 of 15 and explicit-lesson runs 9 of 15.

What the ArchDocs agents did explains most of the misses. We read each run's ArchDocs calls from the filter log and its document mentions from the event log:

- Team archive: all three called task context and opened the teams design doc that records the rule. All three passed.
- Review provenance: both completed runs called task context and opened the design review holding the decisions. One passed and one failed on migration order.
- Review pricing: two of three made no ArchDocs call at all. The third read the design doc and built no separate upload setting.
- Release check: two of three called task context and the relevant design doc appeared in the response, but neither opened it. The third made no ArchDocs call.
- View refresh: all three called task context and opened the lesson's design doc. Every setup passed this task.

Across the five valid tasks, 11 of 14 completed ArchDocs runs called task context, and 13 of 17 across all six tasks. A run that never asks gets nothing from ArchDocs, so those runs stay in the ArchDocs denominator as product failures. Several ArchDocs agents also tried to write Work Log entries, as the plugin instructs, and the read-only filter refused them.

Cost went up with ArchDocs. Among runs with recorded usage, the median input was about 2.27 million tokens per ArchDocs run (14 of 17 runs recorded), 1.53 million per baseline run (16 of 18) and 1.62 million per explicit-lesson run (17 of 18), most of it cache reads. Median wall time, including the timeouts, was 452 seconds for ArchDocs, 351 for the baseline and 401 for explicit lesson.

### [Worked example: team archive](#worked-example-team-archive)

The brief, identical for every setup:

Add POST /teams/{id}/archive for organization administrators. Archived teams should disappear from active team lists and pickers, while documents and spaces previously associated with the team remain available. Return a useful error when the operation cannot be completed.

The recorded decision lives in DD-0027, a draft design doc for an overhaul of team management. An entry in its Work Log, from a re-audit of the team code, recorded that deleting a team that approvals depend on blocks those approvals. The plan's approach requires checking approver-pool dependencies before a team is archived or deleted. Neither statement exists in either repository.

In the delivery test, task context pointed at DD-0027 for this lesson without stating the rule. In one ArchDocs trial run, the agent called task context once with a paraphrase of the brief, read DD-0027's summary and then its full spec, read one other design doc, and fetched DD-0027's Work Log handoff. The filter logged the size and hash of each response but not its text, so we can't show the exact context that run received.

That run's change looks up enabled approval policies that name the team as their approver team before archiving. If one exists, it refuses with HTTP 409 and a message telling the administrator to replace the approver team first, and it adds a test for that case.

The grader seeds a Postgres database with a document whose approval rule names the team as its only live approver team, including a document pinned to a change-type revision, and confirms that someone can approve it. It then requests the archive. A pass requires an error status of 400 or higher with a message about approvers or the dependency, an unchanged database, and an approver who can still approve. All three baseline runs got HTTP 200: the archive went through. All three ArchDocs runs and all three explicit-lesson runs refused and left the data unchanged. The grader stops at the accepted request, so for baseline runs it didn't separately measure the state afterwards.

### [Analysis history and statistics](#analysis-history-and-statistics)

We didn't choose a statistical test before the trial. The first analysis restricted to three "readable tasks" (ArchDocs 3 of 9, baseline 0 of 9, one-sided Fisher p = 0.10). We chose that filter after seeing results, and it excluded the review-provenance row, so we abandoned the filter. The primary analysis is every valid task, reported as the counts above.

A one-sided Fisher exact test on the pooled totals (7 of 14 against 3 of 15) gives p = 0.095. That test treats the 29 runs as independent. They aren't: runs of the same task share its difficulty and its grader, and outcomes were almost identical within each task, so the effective sample is closer to five tasks than to 29 runs. The test also ignores the blocked run order. We report the pooled p-value as exploratory only. Excluding the task every setup passed gives 4 of 11 against 0 of 12 (p = 0.037). That exclusion was also decided after seeing results, so it's exploratory too and isn't an alternative headline. Team archive alone is 3 of 3 against 0 of 3 (p = 0.05), the smallest p-value three runs per setup can produce. It shows a consistent difference on one task, not a general rate.

## [What this supports](#what-this-supports)

On one task, a decision recorded only in ArchDocs reached the agent and prevented a specific bug in every run. One more ArchDocs run passed a task no other setup passed, with more source detail than the explicit-lesson setup had. On the other three tasks, ArchDocs made no difference, and the misses mostly came from agents that didn't call task context or didn't open the record it pointed to.

It doesn't support a general claim about correctness or productivity. That would need tasks drawn from ordinary work, held-out retrieval cases, a baseline with repository copies of the same plans, and a test chosen in advance with the task as the unit. Other limits: one organization's records, one solver model per stage, one judge model, graders written by an AI model, and records partly written by agents that were using ArchDocs at the time. We built ArchDocs and ran the study ourselves.

## [Evidence and what isn't public](#evidence-and-what-isnt-public)

The records behind these numbers are in a private benchmark repository and our private ArchDocs workspace: the draw and screening outputs, task briefs, graders and reference implementations, the blind keys and grader diffs, and for every run the prompt, patch, event log, filter log, token usage and grades. They include our private source code and internal design docs, so we haven't published them, and readers can't reproduce the study from what's here. Publishing a sanitized bundle (briefs, graders run against a public fixture, per-run grades and call traces without response text) is the gap we'd need to close first.

## [Next experiments](#next-experiments)

- Held-out retrieval: judge task context on lessons recorded after the tuning, to find out whether delivery holds up on cases it wasn't tuned on.
- Same-passages baseline: give the baseline the exact passages ArchDocs retrieved, at task start, to separate retrieval quality from the agent's ability to use supplied context.
- Repository-copy baseline: export ArchDocs-only specs and plans into the repository as files, to find out whether ArchDocs adds anything when the same plan is searchable.
- More tasks, fewer runs each, with the analysis fixed in advance and the task as the unit, to estimate how often this happens across ordinary work.
- Consultation: measure how often agents call task context and open what it points to, before and after changes to the plugin's instructions.
- Process lessons: a judged trial on agent plans and reports for the 30 lessons a code test couldn't grade.

**The ArchDocs team** 25 min read

More in Coding agents, context and verification

Start