Two months ago I wrote here that my benchmark said 98.3% and the number was worthless. It was true. It just measured the wrong thing.
Since then I have been working on the same question: what is the honest number for an AI memory? I now think it is not the hit rate. It is how often a memory the system surfaced on its own actually got used afterwards.
That difference sounds small and it is not. A hit rate measures whether the search found something. A usage rate measures whether it helped anyone. Only the second one can come out against you.
Version 1.0 measures both and shows them. Here is one week from my own vault.
In seven days, 21,569 events. The memory wrote 4,136,535 tokens into my context windows — and that is a lower bound, because fifteen entries carry no size and count as unknown rather than as zero.
Now the part that hurts. Of the memories it surfaced in the top band, the ones it considered required reading, 2,326 were put in front of the assistant. 35 of them were actually loaded. That is 1.5%.
Read the other way: 1,670 recalls came back with hits, and 68 of them led to a load. 4.1%.
I could stop here and call it a bad system. But there is a second number that changes the picture. Of the 35 that did get loaded, 34 measurably changed what happened next. A 97% usage rate.
So the selection is not the problem. The volume is. My memory is good at picking the right thing and bad at keeping quiet about everything else, and without this view I would have had no way of telling those two failures apart. I would have tuned the ranking, which was never broken.
Every number above can come out against my own product. That is the point. A memory you cannot audit is not a memory, it is a promise — and the tools in this category are full of promises.
It is open source under MIT and runs locally: https://github.com/n0mad-ai/bastra-recall The numbers are from my own vault over the last seven days. Anyone who installs it sees their own, on the same screen.