I published a result that said my tool scored 34.5% and another scored 5%. A reader on Reddit asked two questions about how I counted. I went back, fixed the method, tested 40 more commits, and the result didn't hold.
This is what went wrong, and how I'd run this kind of test now.
I build EngramGraph, a code graph. I wanted to know if a code graph can tell an AI agent which other files need to change when a function changes. So I compared it with code-review-graph, an open source tool with about 32k stars on GitHub.
The method was simple. Take a real pydantic commit. Go back to the parent commit and index it. Ask each tool who calls the changed functions. Compare the files it names with the files the commit really changed.
I froze the method in a file before running it, and I ran 20 commits. That felt rigorous. It wasn't enough.
The first question: when a tool declines to answer, is that a miss, or is it dropped from the denominator?
It was a miss. An empty prediction scored 0 for that commit, and I averaged across commits. code-review-graph declined 18 of 42 lookups, so most of its score came from not answering. My headline was mostly measuring how often each tool answered, not whether its answers were right.
The second question: what's the ceiling? Many files change together without any call between them, so 34.5% means something different against a ceiling of 40% than against 100%. I hadn't measured one.
They also suggested showing per-commit numbers next to the pooled ones. I'm grateful for all three. None of this needed graph expertise, just someone reading the method carefully.
I had described those 18 lookups as "returns nothing". Reading the source showed that was wrong.
With a bare function name, code-review-graph searches names, paths and signatures as full text. Underscores count as word breaks, so rebuild_model_fields also matches test_rebuild_model_fields_preserves_description. When more than one node matches, it stops and returns a ranked candidate list, asking you to retry with a full name. That's a reasonable design for an agent. My script treated the reply as empty.
My first run said one tool was far ahead. Across all 60 commits, with 171 files to find:
So the honest summary depends on whether your agent retries, and I haven't measured that. If it does, the two are about level and code-review-graph is more accurate.
Calls also explain less than I assumed. Only about 30% of the co-changed files mention a changed function at all, so that's a rough ceiling for any tool working from calls.
I've stopped using these tests for that kind of comparison. I still run them, because they're the best way I have to find what to fix.
The most useful thing that came out of this was a signal neither tool uses: files that historically change together. Added to EngramGraph's callers on the 40 new commits, it found 29 files instead of 19, at a cost of 6 points of accuracy. It's only been checked on one repository.
All tables use the same scoring: files predicted, files correct, share correct when it answered, share of the files to find, and F1 pooled across commits. "Answered" counts commits with at least one prediction.
Round one, as first published (20 commits, averaged per commit, empty answers score 0):
| Precision | Recall | F1 | |
|---|---|---|---|
| EngramGraph | 0.345 | 0.345 | 0.337 |
| code-review-graph, bare name | 0.05 | 0.05 | 0.05 |
| code-review-graph, retry (added after the run) | 0.15 | 0.145 | 0.139 |
**Round one, pooled** (45 files to find):
| Answered | Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|---|
| EngramGraph | 11 / 20 | 18 | 10 | 56% | 22% | 0.32 |
| code-review-graph, bare name | 1 / 20 | 1 | 1 | 100% | 2% | 0.04 |
| code-review-graph, retry | 4 / 20 | 8 | 5 | 63% | 11% | 0.19 |
Round two, frozen in advance (40 new commits, 126 files to find):
| Answered | Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|---|
| EngramGraph | 24 / 40 | 65 | 19 | 29% | 15% | 0.20 |
| code-review-graph, bare name | 10 / 40 | 16 | 6 | 38% | 5% | 0.09 |
| code-review-graph, retry | 13 / 40 | 33 | 16 | 48% | 13% | 0.20 |
All 60 commits (171 files to find):
| Answered | Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|---|
| EngramGraph | 35 / 60 | 83 | 29 | 35% | 17% | 0.23 |
| code-review-graph, bare name | 11 / 60 | 17 | 7 | 41% | 4% | 0.07 |
| code-review-graph, retry | 17 / 60 | 41 | 21 | 51% | 12% | 0.20 |
Git history, new 40 commits only (top 2 files per commit, using only commits before the parent):
| Predicted | Correct | Right | Found | F1 | |
|---|---|---|---|---|---|
| EngramGraph callers | 65 | 19 | 29% | 15% | 0.20 |
| Git history alone | 80 | 16 | 20% | 13% | 0.16 |
| Both | 125 | 29 | 23% | 23% | 0.23 |
Retries in round two: code-review-graph declined 47 of 78 lookups. Retrying found caller files for 24, found no callers in other files for 8, and had no usable candidate for 15. My script made those retries, picking the candidate in the seed file. A real agent might not retry, or might pick wrong.
Ceiling: 51 of 171 co-changed files mention a seed function by name, from a text search on the parent commit. That's an estimate.
Limits: Python only, one repository, direct callers only. "Changed in the same commit" stands in for "affected", so a real caller that didn't need changing counts as wrong. Tool versions were code-review-graph 2.3.9 and EngramGraph 0.13.0. Its main feature, impact analysis from a diff, wasn't tested. In round one the pooled numbers and retry were added after the run; in round two everything was frozen first. I build EngramGraph, so weigh that.
This article was written with the help of AI, based on my experiment notes and the scoring output.