A few weeks after I published my write-up for grounded-mcp, a reader left two comments under it - the first said the CI-gate framing was strong, then listed what the regression set should contain: superseded versions, permission-denied cases and unsupported questions, reported by slice with the case count next to each pass and fail. The second asked which of the three slices had proved to be the hardest.
If you've already read the first article then you know that two of the three were already in the suite. Permission-denied is the leakage family - an independent hard-zero gate that runs with the thresholds off - and unsupported questions are the abstention family. The third was missing. Nothing within the server knew that one note could replace another, and nothing in the scorecard had a row where that failure could surface. I had built a knowledge server whose trust properties were CI-gated numbers, and one of the properties a real vault values most was not a number at all.
So I measured how bad it was, which you should also do before you believe anything below.
The demo vault is a fictional company handbook: public, internal and restricted zones, twenty-odd notes. I added four more, each a plausible previous version of a note already there. Last year's expenses policy next to this year's. The old deploy runbook next to the current one. An archived product datasheet. Last year's salary bands in the restricted zone. Same titles, same tags, old numbers: meals at £35 a day instead of £40, a twenty-minute manual rollback instead of a ninety-second automatic one, a £199 list price instead of £249.
This is not a contrived vault. If you keep notes in a store like Obsidian for more than a year then you'll have this vault. Then I asked the unchanged v0.2 server twelve questions those notes answer. Read the table as if it were your handbook and your agent.
query rank_cur rank_stale score_cur score_stale cov_cur cov_stale
meal allowance while travelling 2 1 5.30 5.30 0.75 0.75
hotel cost cap when travelling 2 1 7.95 7.95 0.75 0.75
expense claim deadline days 2 1 5.31 5.31 0.75 0.75
friday deploy freeze rules 2 1 9.54 9.54 0.75 0.75
senior engineer salary band (exec) 2 1 10.39 10.39 1.00 1.00
how much does widget pro cost 1 2 4.82 4.77 0.67 0.67
how long does rollback take 1 2 5.13 4.91 1.00 1.00
...
stale ranked first: 8 of 12 stale in the cited top five: 12 of 12
The stale note tied its successor on BM25 score, to two decimal places, in seven of the twelve queries. It tied on term coverage in eleven of twelve. Where the scores tied, the stale note won every time. FTS5 returns tied rows in insertion order, notes are inserted in sorted path order, and "expenses-policy-2025.md" sorts before "expenses-policy.md" because a hyphen comes before a full stop in ASCII. Which version of the policy your agent would have quoted was decided by the character code of a hyphen.
Both abstention gates passed the stale note in twelve of twelve, and that's the finding, and it's worth sitting with. The abstention design in the first article was built to catch a note that does not answer the question. A stale note answers the question. It is relevant, it has the right title, it covers the query's terms, but it is just no longer true - and there is no relevance signal on earth that can see that. If your retriever is scoring relevance, and every retriever is, you have this problem.
Here is what the v0.2 scorecard said with the four notes in place:
retrieval (staff, n=22): hit@1 0.82 recall@5 1.00 MRR 0.91
abstention (n=10): rate 1.00 (target 1.00)
leakage : 0 (must be 0)
redaction : 0 (must be 0)
EVAL GATE: FAIL
- retrieval.hit_at_1 regressed: 1.00 -> 0.82
You could read that as a win. The gate went red. But look at what it is complaining about: a ranking regression in retrieval. recall@5 is still perfect, because the current note is still in the top five. Abstention is still perfect, because the server was confident. And it only went red at all because my golden set happens to list only the current path as an acceptable answer for those queries. The field is documented as "any of these counts". A golden set written the way the documentation invites you to write it would have stayed green.
Shekhar Kadyan at Pipeshub put the general case in one sentence a few weeks before I ran this: "The system was measuring one property and being graded on a different one". My system was measuring rank and being graded on currency. Jason Liu has the practitioner version for aggregates: "You see a recall of 70% and think, 'Not bad.' But that might average across many easy queries". Mine said 1.00 on recall and was averaging across a failure it had no name for.
None of this is new. Mitchell and colleagues wrote in the Model Cards paper in 2018 that "Quantitative analyses should be disaggregated, that is, broken down by the chosen factors". Google shipped sliced metrics in TensorFlow Model Analysis the same year. Slice Finder exists because, in its authors' words, "the overall model performance can fail to reflect that of the smaller subsets". I knew all of that. I still shipped a scorecard with four aggregate lines, because the failure mode I needed a slice for was one I had not seen yet. Shreya Shankar calls this criteria drift: you cannot specify the criterion until you have watched the failure. That explains the miss. It does not excuse it.
I had assumed this was a lexical problem, a quirk of BM25 that hybrid semantic retrieval would soften. The literature says the opposite. Neeraj Yadav, in a paper from June, states it as a structural property: "RAG gives language-model agents access to accumulated knowledge, but it has no model of time". His example is a timeout that changed from 1800 to 3600 seconds: the two facts "differ by one token and sit close together in any embedding model". Then the line I would have paid for: "This is the error RAG cannot avoid by construction: it retrieves both values and has no mechanism to choose". His measured rate, in a synthetic setting, is that "RAG serves the superseded value 15–40% of the time". His follow-up on 707 real GitHub fixes found 36 to 38 per cent. Those are one author's two papers, with a system of his own to sell, so take the percentages as a range from one lab rather than a consensus. An independent study from late September narrows that gap. Ahmed, Poech and Röttger built 317 verified knowledge reversals across medicine, law, software and platform policy, and found that outdated retrieval "flips 30% of Llama and 37% of Qwen answers" that were right without retrieval. A recency-aware re-ranker won back only 4.6 to 10 points, and only when the dates were accurate; with wrong dates it can favour the old evidence. The mechanism, though, is exactly what my table shows. Two versions of a policy have the same term profile for BM25 and near-identical embeddings for a semantic model - so better retrieval actually makes the tie tighter rather than looser.
And the model on the other end of the MCP connection will not save you either. Google's Sufficient Context work found that giving a model retrieved context "paradoxically reduces the model's ability to abstain from answering when appropriate", with Gemma going "from providing incorrect answers on 10.2% of questions with no context up to 66.1% when using insufficient context". A superseded note is the purest form of insufficient context: it looks complete, reads with authority and answers the question with last year's number. If your server serves it, your model will often use it, and it will sound sure.
The obvious fix, and probably the one you would reach for, is a frontmatter field the ranker can see, plus a penalty. I did not do that, and a paper from this month is the reason. Shen, Toyoda and Leung tested five agent-memory systems that mark revoked facts invalid but keep them, across nine policy scenarios and nine models, and found that "no system enforces revocation by default: the revoked fact is returned wherever the revocation label is visible to the retrieval layer, outranks its replacement, and leads agents to the unsafe action". A label the retriever can see is a suggestion. To be fair to labels, the September study found that larger models "switch to the appropriate answer almost perfectly" when they are told explicitly when the old evidence stops applying. But that is the model reading a label after retrieval has already chosen, and Shen's systems show the revoked fact outranking its replacement before the model gets a say. My leakage gate already taught me the same lesson about entitlements in the first article: filtering results after scoring leaks, because the denied content still shaped the ranking. So supersession got the same treatment.
Mark the old note in its frontmatter with the path of the note that replaced it:
---
title: "Expenses Policy"
superseded_by: internal/hr/expenses-policy.md
---
At index build, a note with that key is never inserted. It cannot be scored, ranked, snippeted or cited, at any threshold, because it is not in the candidate set. Read it directly by path and you still get it, flagged superseded: true with the successor path and a warning, so history is reachable on purpose and never by accident. Entitlements are applied first: a stale note in the restricted zone is as invisible to staff as a current one.
I am not claiming to have invented any of this. Dublin Core has had isReplacedBy and replaces for years, DataCite has IsObsoletedBy, and every RFC carries an "Obsoletes:" header. Document control under ISO 9001 is largely the discipline of making sure the obsolete copy is not the one on the wall. What surprised me is how little of that vocabulary has reached the servers people are wiring agents to. When I first looked, in September, three agent-memory stores modelled supersession at all, and only one of them, rohitg00's agentmemory, said in plain words that "Superseded memory versions leave the search indexes". Looking again in November, I found a handful of small markdown and Obsidian servers doing the same: docs-rag-mcp stops returning a note marked superseded, local-knowledge-rag-mcp excludes anything tagged obsolete, and agent-mem-vault leaves superseded records out of search. All of them are recent and none publishes an eval for it. I found no Drive, Notion or Confluence connector with any notion of one note replacing another. Elasticsearch's collapse feature does the opposite of what I did: score everything, then de-duplicate to the newest after. That works, but honestly, a hard-zero gate is easier to prove when the stale note cannot appear in the candidate set in the first place.
The suite now reports by slice. Every golden case belongs to exactly one, named for the failure it can expose, and each row prints its case count next to pass and fail.
slice n pass fail detail
answerable 24 24 0 hit@1 1.00 recall@5 1.00 MRR 1.00 miss 0 false-abstain 0
unsupported 10 10 0 abstention 1.00 false-answer 0
superseded 16 16 0 stale-cited 0 (must be 0) current-hit 12/12 history-readable 4/4
denied 24 24 0 leaks 0 (must be 0)
secret 6 6 0 violations 0 (must be 0)
EVAL GATE: PASS
Three things changed beyond the new row. False abstention is now its own count: an answerable query that abstains used to be folded into a miss, and "no answer when there was one" is a different bug from "the wrong answer", with a different fix. The superseded family is a hard zero, like leakage: the stale path must not surface even with thresholds off, the current path must still be found, and a direct read of the stale path must carry the right pointer. And the gate compares each slice's fail count against the committed baseline, so a slice cannot get worse while the aggregate looks fine. Zhang, Wang and Lei measured that masking in agent evals in June: six local regressions moved the aggregate pass rate by 1.7 to 5.9 points while the matching slice fell by 25 to 91. Their rule for a slice with no cases is the one I wish I had started with: report it as uncovered, "never 1.0".
The n column is not decoration. Evan Miller's paper on error bars for evals argues for a standard error beneath every mean, and warns that related questions make a suite look "much more precise than it actually is". At my scale the arithmetic is blunt: a slice that passes ten of ten has a 95 per cent Wilson lower bound around 72 per cent. Print "abstention 1.00" on its own and you are implying a precision the suite does not have. Print "10 of 10" and your reader can do the sum. For the hard-zero families I do not print a rate at all, because they are not estimates - they are merely assertions with a case count.
Run the same scorecard with supersession support switched off and you get the answer to the reader's question:
superseded 16 0 16 stale-cited 12 (must be 0) current-hit 12/12 history-readable 0/4
EVAL GATE: FAIL
- superseded.stale_cited = 12 (hard zero violated)
Which slice was hardest? Superseded, zero of sixteen, against twenty-four of twenty-four, ten of ten, twenty-four of twenty-four and six of six for the other four. It was not hard in the sense of a low pass rate on a working mechanism. There was no mechanism. The other four slices each had a design behind them: a coverage gate, a per-profile index, a set of secret patterns. This one had nothing, so it scored nothing, and the old scorecard could not even tell me.
Supersession is declared, not detected. The server trusts what your vault says. If two notes are versions of each other and nobody has marked the old one, it is served. The slice exists so you can measure how many of those you have, not so the server can guess on your behalf. Yadav's GitHub study puts clean atomic replacement at about 18 per cent of real fixes. The superseded_by key models exactly that clean case. It does nothing for a note that is half right, or for the RFC-style "Updates:" relationship where the old document stands with amendments.
The numbers are small. Twelve queries and four reads is enough to prove the failure mode exists and to gate a regression. It is not enough to estimate a rate, which is why I have not printed one. And the real-world evidence that this failure costs money is thinner than I expected. The one properly sourced case is Moffatt v. Air Canada, where a tribunal in 2024 refused to accept the airline's argument that its bereavement web page should outweigh its chatbot: "[Air Canada] does not explain why the webpage titled 'Bereavement travel' was inherently more trustworthy than its chatbot". Two sources on one site disagreed and the airline paid. The record doesn't say that the chatbot retrieved a stale document, so neither will I.
Finally, "none found" is not "none exists". I searched for MCP servers that treat stale content as a first-class concept and ship evals as a CI gate, and, as of November 2026, found none that do both.
grounded-mcp v0.3 is on GitHub, MIT. Add superseded_by to any note you have replaced, run the evals, and read the superseded row. Then do the thing that produced this article: write down the failure you have never seen, put a case for it in your vault, and see which row it lands in. If it lands in the wrong row, that is your next slice.
Hybrid semantic retrieval is now v0.4, and it lands only with its eval delta published against this baseline, per slice. Given what Yadav measured, I expect the superseded row to be the one it does not move.