cd /news/ai-agents/a-prompt-can-t-see-the-second-copy-o… · home › topics › ai-agents › article
[ARTICLE · art-170195] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

A prompt can't see the second copy of a bug. A program can.

A developer has built Ockham, a Claude Code plugin that combines a seven-rule prompt (314 words) with a project index to improve coding-agent behavior on a 44-task benchmark. Rules alone raised hidden-check pass rates from 87/110 to 105/110 and test-writing from 0/70 to 60/70, but no wording fixed duplicated bugs in other modules; the index, which flags when an edited line also appears in another file, fixed both copies in all 70 sessions where the note fired, with no false warnings across 546 indexed sessions. The developer reports the index shortens sessions rather than lowering cost, since savings per call are offset by more calls.

by read5 min views1 publishedNov 16, 2026

A coding agent got a ticket: a customer with exactly 100 points can't redeem the 100-point reward. It found redeemable() in src/loyalty/redeem.js, changed a > to >= and reported it fixed. Four turns, one line, no test.

The same comparison was also in src/emails/points.js. It stayed wrong.

That wasn't bad luck. On that ticket, with no tool, the second copy was fixed in 0 sessions of 5. With one note at the moment of the edit, in 15 of 15.

I'm building Ockham, a plugin for Claude Code, to find out what you can and can't get from a coding agent by telling it things. These are the measurements so far. Most of what they say applies whether you use Ockham or not.

Every number below is one headless agent session on a fresh copy of a project of about forty source files. The agent gets a ticket and nothing else. What it leaves behind is scored by checks it never saw: whether it works, whether it holds on input the ticket left out, whether it reached every place the bug was in, whether it reused what the project already had, whether it left a test where the logic needs one.

There are 44 tasks: 39 to tune on and 5 held out. Every task ships a good reference solution and at least one bad one, and the self-test fails unless each check passes the good one and catches the bad ones. The reference arm is ponytail (MIT), the ruleset that made "make the agent write less" popular. Ockham is written from scratch.

Seven rules, 314 words, tuned one change at a time. On the 22 tasks they were tuned on, five sessions a task:

No tool ponytail Ockham rules
Passed every hidden check 87/110 86/110 105/110
Held on input the ticket left out 103/110 100/110 110/110
Reached every place the bug was in 14/25 11/25 20/25
Left a test where the logic needs one 0/70 7/70 60/70
Cost per session $0.030 $0.036 $0.054

The first rule says which half wins: "the smallest change that does the whole job. If they pull apart, the whole job wins." It's there because of what a plain "do less" rule does. In the baseline, the agent with that rule reused everything, then left out part of the job and reported the omission as a saving: a CSV export with no quoting, five times out of five.

What no wording fixed was the copy of a bug in another module. One wording made it worse: the agent started saying it had looked for other copies. Four sessions of five closed with a search or a check that the project contradicts, and three of those had run nothing.

If you keep one thing from this post: a rule can't make an agent find what it doesn't know exists. That's a job for a program.

So Ockham keeps an index of the project. When a session starts, the agent is told which helpers the files share and where the tests are. When an edit replaces a line that another file also has, it's told which file and which line.

On six tasks nothing had been tuned on:

No tool Rules alone Rules and index
Passed every hidden check 21/30 26/30 27/30
Reached every place the bug was in 9/15 14/15 15/15
Cost per session $0.030 $0.050 $0.047
Time per session 16 s 25 s 21 s

Across every run so far, the note about a copy went to 70 sessions, all on tasks where a bug is written twice, and all 70 fixed both places. No false warning in 546 sessions with the index.

The part I didn't expect: the index doesn't make a session cheaper. It makes it shorter. A shorter list and no list at all were tried, and they cost the same. What the text saves on every call, the agent spends on more calls.

When the agent means to stop, Ockham compares the files the task changed with what they were. Seven checks: a dependency the request didn't name, a second version of something another file exports, a copy that's still wrong in another file, a check the code used to make and no longer makes, risky logic with no test, tests not run after the last change, and a summary that says the tests pass when none ran.

If one fires, the agent is told once and goes on.

It costs more: about $0.05 a session against $0.03 with no tool. Most of that is the tests it asks for. Without the rule that asks for them, a session costs a quarter less and passes the same hidden checks. I kept the rule, because "checked" is the promise. OCKHAM_TESTS=off leaves it out.

Still open:

In Claude Code, with Node.js 20 or later:

claude plugin marketplace add Octonove/ockham
claude plugin install ockham@ockham

Every table above, the harness and the 44 tasks are in the repo: https://github.com/Octonove/ockham

If you run the benchmark on your own setup and get different numbers, I want to see them.

── more in #ai-agents 4 stories · sorted by recency
── more on @ockham 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/a-prompt-can-t-see-t…] indexed:0 read:5min 2026-11-16 · —