With only our rules files, coding agents archived a team documents needed as approvers. With ArchDocs, they found the recorded decision. Most other tasks tied.
- Author
- Aylon
- Published
- Reading
- 9 min read
We asked a coding agent to add an archive endpoint for teams in ArchDocs. Three runs had only our repository and its rules files. Each built a working endpoint, and each let an administrator archive a team that a document's approval rule named as its only approver team. An earlier audit of our team code had found that removing such a team blocks approvals on those documents. Three runs with ArchDocs connected called task context, opened the design doc where that audit was recorded, and built a check that refuses the archive while a document still depends on that team for approvers.
ArchDocs keeps the decisions, findings and Work Log entries a team records while it works. When an agent starts a task, it can ask ArchDocs for task context: the records that match the task, the repository and the files it's about to touch. The design doc for that teams overhaul was still a draft, and the audit's finding existed nowhere in the repository.
That 3 of 3 against 0 of 3 is the clearest result from two days of testing on our own work, November 5 and 6, 2026. The first round didn't produce anything like it: in the pilot, all ten runs passed, five per setup. Across the five tasks that produced a valid trial, ArchDocs changed the outcome on two and made no difference on three. On two of those three, most ArchDocs runs either never asked ArchDocs anything or didn't open the record it pointed to. The full method, grader history and analysis are in How we tested ArchDocs against agent rules files.
What each setup had #
The baseline was our own setup as we actually run it: the repository at a fixed commit, its git history and our rules files. The ArchDocs setup had all of that plus ArchDocs. The pilot used Claude Code with Sonnet, which reads CLAUDE.md. The later coding trial used Codex, which reads AGENTS.md, so we generated an AGENTS.md beside each CLAUDE.md, identical except for one Claude-only section we removed. Every trial result below is a Codex result.
A baseline agent could read any file and search the history, so a lesson written in a spec or a commit message was available to it if it went looking. What it got automatically at the start of a task was only its rules files.
Some of our decisions exist only in ArchDocs because we now write specs there. A team without ArchDocs would probably keep those as repository files, so we planned to give the baseline a repository copy of each one. The pilot did this. The coding trial didn't. Three of the five valid trial tasks, team archive and both review uploads, had lessons with no copy anywhere in the repository, so those results show agents finding decisions only ArchDocs had. They don't show ArchDocs beating a repository that keeps the same plan as a file.
The pilot tied #
The pilot was five tasks, each built around something we'd learned the hard way. We removed the written lessons and their source files from a fixed copy of our monorepo, then gave the same sources back to each setup: path-scoped rules files for the baseline, imported documents for ArchDocs. Each task ran once per setup.
All ten runs passed the feature test and the lesson test, because the code next to every task already did the right thing and every agent copied the nearest helper. Each side also had a weakness. The agents read files through shell commands, so the baseline's path-scoped rules never loaded. Task context came back empty in four of five ArchDocs runs because it matched the task description too literally, which we fixed before going further. ArchDocs runs used more input tokens in all five pairs, because its tool definitions and instructions come along.
Most lessons are already in the code #
Before running more tasks I wanted to know how often there's anything for ArchDocs to add. Codex models extracted 319 lessons, each with its source, from our rules files, 379 fix commits, our specs and plans, our plugin, and 53 design docs in our ArchDocs workspace. About a third were already in a rules file. Nearby code showed 187 of them outright and 47 partly. 18 were confirmed to exist only in ArchDocs, and up to 9 more might, pending a check of one repository we didn't examine. The methodology post has the full breakdown and what the extraction can't tell you.
So in our case ArchDocs rarely knows something the repository doesn't. Most lessons sit in one of hundreds of specs, plans and commit messages, and an agent only gets them if it knows where to look.
Getting the lesson to the agent #
The next test measured what reaches the agent before it writes any code. We took 85 lessons the code doesn't fully reveal, wrote a plain task description for each, and compared what the rules files deliver for those paths with what one task-context call returns. A separate model graded each response blind: it conveys the lesson, it points to the record that holds the lesson, or neither.
The rules files conveyed all 24 controls, the lessons already written in a rules file, and almost none of the other 61. For those 61, task context fully conveyed 10 lessons and pointed at the right record for 31 more. With both sources together, 35 of the 85 lessons were conveyed, against 25 for rules files alone.
We also used these 85 cases to find and fix retrieval problems, including answered design doc questions that weren't indexed and a meaning-based matcher that made results worse until we capped it. So these numbers show how far the fixes moved the cases we tuned on. We haven't yet measured task context on fresh cases.
What the agents built #
Being handed the right record doesn't mean an agent builds the right thing, so the last test graded code. We screened 64 candidate lessons for tasks where nearby code doesn't give the answer away and an automated test can tell compliance from a plausible mistake. Six survived, including team archive, a spare we added when the controls produced no usable tasks. One turned out to be invalid because the grader expected a server endpoint the task never asked for. In the trial, each task ran three times in each of three setups: the baseline, the baseline plus ArchDocs, and an explicit-lesson setup with the lesson written into AGENTS.md as one or two sentences. That last setup shows whether a task depends on the lesson at all. One ArchDocs start was lost to a setup error, so that cell has two runs.
The table counts runs that passed both the feature test and the lesson test, for all five valid tasks.
| Lesson | Baseline | ArchDocs | Explicit lesson |
|---|---|---|---|
| Archiving a team must check whether documents depend on it for approvers | 0/3 | 3/3 | 3/3 |
| Uploaded agent reviews keep who sent them, and unchecked citations stay labelled as reported | 0/3 | 1/2 | 0/3 |
| Agent review uploads are free on every plan, behind their own setting | 0/3 | 0/3 | 3/3 |
| A release check passes a scenario on two of three tries | 0/3 | 0/3 | 0/3 |
| A save refreshes each affected view once and closes the editor without waiting for it | 3/3 | 3/3 | 3/3 |
These are five tasks we picked because they could show a difference, run three times each. They aren't a sample of ordinary engineering work.
The second row came from a design review's answered questions for a feature that wasn't merged yet. The ArchDocs agents read those decisions in full, while the explicit-lesson setup got a two-sentence summary, so this row doesn't show that retrieval beats giving the agent the same material. One ArchDocs run passed, after a grader repair we made while reading ArchDocs failures: the grader left out a model field the design doc requires. We fixed it without changing any assertion and regraded every upload patch without knowing which setup produced it. The other ArchDocs run dated its database migration so that it runs before a table it depends on exists, which fails on a fresh database. We counted that as a failure. With the migration renamed, it still failed one of the two lesson checks, because it kept a verification timestamp supplied by the up client.
The last row is a screening miss. Our screen judged that nearby code didn't show the pattern, but every agent in every setup got it right.
Where it failed #
On the third row, the explicit lesson worked and ArchDocs didn't. With the explicit lesson, all three runs built the separate upload setting and the free-on-every-plan rule. With ArchDocs, two of three runs never asked ArchDocs anything. The third read the design doc and still didn't build the setting.
On the release-check task, two ArchDocs runs were shown the right design doc in their task context, and neither opened it. Even the explicit lesson didn't get any run to pass that one.
Across the five valid tasks, 3 of the 14 completed ArchDocs runs never called task context. In the product, an agent that doesn't look gets nothing from ArchDocs, so I've counted those runs as ArchDocs failures rather than dropping them. ArchDocs runs also cost more. Across all six trial tasks, among runs where usage was recorded, they used a median of about 2.3 million input tokens, mostly cached, against about 1.5 million for the baseline.
What this shows #
On one task, a decision recorded only in ArchDocs reached the agent and prevented a specific bug. Across five selected tasks ArchDocs helped on two, and where it missed, the agent mostly didn't ask or didn't open what it was shown.
That gives us our next work. We need agents to call task context reliably and open the record it points to, and we need task context to return the passage that states the rule. We also want a fresh set of cases we haven't tuned on, and a baseline that gets the same source passages ArchDocs retrieved, so we can tell retrieval apart from what the agent does with the text.
If you want to try the same setup on your own decisions, connect your coding agent and run task context before your next change in an area where you've already recorded one. Task context is available on Team and above. Aylon 9 min read
More in Coding agents, context and verification
Start