A coding agent got a ticket: a customer with exactly 100 points can't redeem the 100-point reward. It found redeemable() in src/loyalty/redeem.js, changed a > to >= and reported it fixed. Four turns, one line, no test.
The same comparison was also in src/emails/points.js. It stayed wrong.
That wasn't bad luck. On that ticket, with no tool, the second copy was fixed in 0 sessions of 5. With one note at the moment of the edit, in 15 of 15.
I'm building Ockham, a plugin for Claude Code, to find out what you can and can't get from a coding agent by telling it things. These are the measurements so far. Most of what they say applies whether you use Ockham or not.
Every number below is one headless agent session on a fresh copy of a project of about forty source files. The agent gets a ticket and nothing else. What it leaves behind is scored by checks it never saw: whether it works, whether it holds on input the ticket left out, whether it reached every place the bug was in, whether it reused what the project already had, whether it left a test where the logic needs one.
There are 44 tasks: 39 to tune on and 5 held out. Every task ships a good reference solution and at least one bad one, and the self-test fails unless each check passes the good one and catches the bad ones. The reference arm is ponytail (MIT), the ruleset that made "make the agent write less" popular. Ockham is written from scratch.
Seven rules, 314 words, tuned one change at a time. On the 22 tasks they were tuned on, five sessions a task:
| No tool | ponytail | Ockham rules | |
|---|---|---|---|
| Passed every hidden check | 87/110 | 86/110 | 105/110 |
| Held on input the ticket left out | 103/110 | 100/110 | 110/110 |
| Reached every place the bug was in | 14/25 | 11/25 | 20/25 |
| Left a test where the logic needs one | 0/70 | 7/70 | 60/70 |
| Cost per session | $0.030 | $0.036 | $0.054 |
The first rule says which half wins: "the smallest change that does the whole job. If they pull apart, the whole job wins." It's there because of what a plain "do less" rule does. In the baseline, the agent with that rule reused everything, then left out part of the job and reported the omission as a saving: a CSV export with no quoting, five times out of five.
What no wording fixed was the copy of a bug in another module. One wording made it worse: the agent started saying it had looked for other copies. Four sessions of five closed with a search or a check that the project contradicts, and three of those had run nothing.
If you keep one thing from this post: a rule can't make an agent find what it doesn't know exists. That's a job for a program.
So Ockham keeps an index of the project. When a session starts, the agent is told which helpers the files share and where the tests are. When an edit replaces a line that another file also has, it's told which file and which line.
On six tasks nothing had been tuned on:
| No tool | Rules alone | Rules and index | |
|---|---|---|---|
| Passed every hidden check | 21/30 | 26/30 | 27/30 |
| Reached every place the bug was in | 9/15 | 14/15 | 15/15 |
| Cost per session | $0.030 | $0.050 | $0.047 |
| Time per session | 16 s | 25 s | 21 s |
Across every run so far, the note about a copy went to 70 sessions, all on tasks where a bug is written twice, and all 70 fixed both places. No false warning in 546 sessions with the index.
The part I didn't expect: the index doesn't make a session cheaper. It makes it shorter. A shorter list and no list at all were tried, and they cost the same. What the text saves on every call, the agent spends on more calls.
When the agent means to stop, Ockham compares the files the task changed with what they were. Seven checks: a dependency the request didn't name, a second version of something another file exports, a copy that's still wrong in another file, a check the code used to make and no longer makes, risky logic with no test, tests not run after the last change, and a summary that says the tests pass when none ran.
If one fires, the agent is told once and goes on.
It costs more: about $0.05 a session against $0.03 with no tool. Most of that is the tests it asks for. Without the rule that asks for them, a session costs a quarter less and passes the same hidden checks. I kept the rule, because "checked" is the promise. OCKHAM_TESTS=off leaves it out.
Still open:
In Claude Code, with Node.js 20 or later:
claude plugin marketplace add Octonove/ockham
claude plugin install ockham@ockham
Every table above, the harness and the 44 tasks are in the repo: https://github.com/Octonove/ockham
If you run the benchmark on your own setup and get different numbers, I want to see them.