{"slug": "i-ranked-my-ai-s-memories-by-age-every-benchmark-floor-broke", "title": "I ranked my AI's memories by age. Every benchmark floor broke.", "summary": "A developer building an agent memory server tested ranking stored lessons by age and found every benchmark floor broke: right-answer-at-rank-1 fell to 76.9% against a 90% floor, mean reciprocal rank to 88.5% against 94%, nDCG@5 to 92.0% against 96%, and the external set's rank-1 hit rate to 67.9% against 78%, forcing the pull request to be closed. The developer concluded that age is a property of the note while truth is a property of the fact, so an explicit supersede link should outrank a later append, and noted that only 2 of 735 stored lessons carried a replacement link, pointing at each other.", "body_md": "Bob Smith wrote [an article on agent memory](https://dev.to/bob_smith_eaea75d3ee1cba3/your-agent-doesnt-have-memory-it-has-amnesia-and-its-an-architecture-bug-178f) that argues for an append-only log with a projected view on top. In the comments he asked me something I owe a long answer to:\n\nHave you hit the case where a supersede and a later append disagree, and if so, which wins?\n\nShort answer: the explicit one. Being later wins nothing by itself.\n\nThe long answer starts with an experiment that failed.\n\nAgent memory goes stale. A note says \"deploys go through script X\". Three weeks later script X is gone, and the note is still there, sounding exactly as sure as it did on day one.\n\nSo the obvious move is to let age count. Older notes rank lower, fresh ones rise. Most memory designs I've read reach for some version of that. So did I. I changed the ranking in my memory server: the older a lesson, the further down it goes.\n\nMy server has a benchmark that runs on every pull request. It's a set of questions with known right answers, over two collections: my own, and an external one shaped differently. Each metric has a floor. Dropping below it blocks the merge.\n\nWith age in the ranking:\n\n[\"Right answer at rank 1 fell to 76.9 % against a floor of 90 %, mean reciprocal rank to 88.5 % against 94 %, nDCG@5 to 92.0 % against 96 %, and on the external set rank 1 fell to 67.9 % against 78 %.\"](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj3ttz481rrivfih34ile.png)\n\nEvery floor broke. I closed the pull request.\n\nThe right answers were often weeks old. That's what useful memory looks like: the deploy trap you hit in July is exactly the thing you need in October. The age penalty pushed those answers under fresh notes that were merely related.\n\nMy mistake was using age as a stand-in for \"no longer true\". They're properties of different things. Age belongs to the note. Whether it still holds belongs to the fact. A lesson can be two days old and wrong, or two years old and right, and a timestamp can't tell those apart.\n\n [\"Two notes. One written two days ago, wrong since yesterday because the script it names was deleted. One written four months ago, still true. Ranking by age asks the clock; a correction that names its target asks the fact.\"](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs9auzqc9bzu1g3kwon0p.png)\n\nWhen two notes disagree, this is what decides it in my server, in plain terms:\n\nSo, Bob: an explicit supersede beats a later append. A later append on its own is one more voice.\n\n\"Explicit beats implicit\" has one catch. Someone has to write the explicit part.\n\nIn September I counted my own store. Of 735 lessons, **2** carried a replacement link. And those two pointed at each other.\n\n [\"735 grey dots, two red ones linked to each other: 2 of 735 lessons carry a link to the lesson they replace.\"](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4fcun33kwlnvl9f4xuz3.png)\n\nAgents correct themselves constantly. They almost never say what they're correcting. When I looked into why, the answer was plain: the agent writing a correction rarely has the old note in front of it. And word overlap won't find the pairs for you. Real correction pairs share about as many words as unrelated lessons do.\n\nSo a design that waits for the writer to declare a supersede waits forever.\n\nThat's where Bob's other idea comes in: **supersede-by-actor**. A passing test that contradicts a model's note, or a human review that rejects it. Those actors know exactly which belief they just overturned. They're the ones who should write the link. I have half of it: a failed check already sits on top of the lesson it failed. The link from the check to the corrected version isn't written yet. That's the next piece.\n\nThis works on any agent memory: a CLAUDE.md, a folder of notes, a vector store.\n\nIf b is much smaller than a, your memory settles every conflict by whatever the reader happens to see first.\n\nI'd like to see your a and b. Mine, counted across the whole store: 2 explicit links in 735 lessons.\n\nBob's question came out of a longer thread under Edward Izgorodin's [Everything It Remembers Has the Same Authority](https://dev.to/izgorodin/your-agent-doesnt-need-more-memory-it-needs-to-know-what-its-allowed-to-believe-22j7). If you care about this problem, the comments there are worth the time. Thanks to [@bob_smith_eaea75d3ee1cba3](https://dev.to/bob_smith_eaea75d3ee1cba3) for asking the question this post answers, and to [@izgorodin](https://dev.to/izgorodin) for the thread it grew out of.\n\nI build **cachly** — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations; cachly remembers **your system**: the bug you fixed, why you chose Postgres, the deploy step that always breaks. Every assistant reads the same memory, and every lesson carries the name of whoever learned it.\n\n**Try it, in increasing order of commitment:**\n\n`npx @cachly-dev/mcp-server@latest demo` in any git repo. It reads your log locally and prints what an assistant would already know about the project. After npx fetches the package, the command makes no network calls.`/plugin marketplace add cachly-dev/cachly-mcp`, then `/plugin install cachly-brain@cachly`.` npx @cachly-dev/mcp-server@latest autopilot` writes the MCP configuration for whichever assistant you use.", "url": "https://wpnews.pro/news/i-ranked-my-ai-s-memories-by-age-every-benchmark-floor-broke", "canonical_source": "https://dev.to/heinrichneb/i-ranked-my-ais-memories-by-age-every-benchmark-floor-broke-2obm", "published_at": "2026-11-09 23:39:05+00:00", "updated_at": "2026-11-09 23:53:10.652684+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence"], "entities": ["Bob Smith", "dev.to"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-ranked-my-ai-s-memories-by-age-every-benchmark-floor-broke", "markdown": "https://wpnews.pro/news/i-ranked-my-ai-s-memories-by-age-every-benchmark-floor-broke.md", "text": "https://wpnews.pro/news/i-ranked-my-ai-s-memories-by-age-every-benchmark-floor-broke.txt", "jsonld": "https://wpnews.pro/news/i-ranked-my-ai-s-memories-by-age-every-benchmark-floor-broke.jsonld"}}