# I let a RAG index rot for 13 months. It grew 5x, 90% failed a check > Source: > Published: 2026-07-30 15:01:09+00:00 [← ragproof.io](/) # I let a RAG index rot for 13 months. It grew 5x, 90% failed a check. *A reproducible teardown of what happens to a vector index after 13 months of real document churn, across pgvector, Qdrant and Chroma.* ## Why I built this I build, consult and teach in the GenAI domain, and I'm an active AWS Community Builder in AI Engineering category. One of my friends came to me with some version of "Why retrieval quality of our in-house RAG degrades over time?" - and I didn't have a satisfying answer other than "let me look". The root cause was removal of a bunch of docs belonging to another track, and this situation inspired me to write this post. I built a version of the worst case on purpose: took a real corpus with real edit history, replayed a year of it the way real ingestion pipelines actually behave (including their bugs), and then measured exactly what's wrong with the result. Engine by engine, against a ground-truth ledger that knows what *should* be true at every point in time. The corpus is the [GitLab Handbook](https://about.gitlab.com/handbook/) - a real company wiki, thousands of Markdown pages, years of git history, CC BY-SA 4.0 licensed (per GitLab's own ["Our Handbook is open source: here's why"](https://about.gitlab.com/blog/our-handbook-is-open-source-heres-why/), not the repo's boilerplate `LICENSE` file, which is silent on content). It reads exactly like the internal wiki or Confluence space most RAG buyers actually index. On top of it: 100 clearly-labeled synthetic employee records (Faker, seed=42, not real people) for a GDPR erasure demo, since a real company handbook by itself has no personal data to erase. ## Method **T0 → HEAD:** picked a commit ~13 months before today (`d9dfa4eb7a` , 2025-06-02) as the starting index, then walked**4,742 real first-parent commits** forward to HEAD (`6a7e263deb` , 2026-07-10, the day I started the experiment) - actual edits, deletes, and renames from a real repository's history, not simulated churn.**One embedding pass, three engines:** every chunk is embedded exactly once (BGE-M3, fp32, HF revision`5617a9f61b` ,`sentence-transformers` pinned to`5.6.0` ) and the*identical*vectors are upserted into pgvector (HNSW, cosine), Qdrant (cosine), and Chroma (cosine) - all on documented defaults, no engine given the other's tuning. Docker image digests, client library versions, and the master seed (42) are all pinned and published in the repo.**The ingestion bugs are on purpose.** Production RAG post-mortems consistently blame ingestion and change-detection, not the model, for index rot. I modeled four canonical failure modes on labeled subsets of the real churn:**(A) Update-without-delete**- re-embed and upsert on edit, but never delete the old chunks. The single most common real bug, and the dominant driver of everything below. Applied to 48.44% of update events.**(B) Async/eventually-consistent deletes**- issue the delete call but don't wait for it to actually take effect before considering the job done.**(C) Full re-ingest duplication**- one mid-timeline "someone re-ran the whole pipeline" event that re-upserts everything under new ids.**(D) Unhandled renames**- index the new path as a new doc, leave the old path's chunks orphaned.- Plus a **GDPR erasure demo**: Article-17-style "delete this person" requests for 40 of the 100 synthetic employees, issued via each engine's normal delete API, with no manual VACUUM/optimize run afterward - because that's what real teams do: call the delete API and assume it's handled. Every single mutation - real or injected-bug - is logged to a SQLite ledger with the *expected* index state per engine. That ledger is published alongside the repo, which is what makes every number below checkable by someone other than me. ## Findings Before the percentages, the shape of it: the index went from **33,286 chunks at T0 to 166,947 live chunks** at HEAD. Five times the size after 13 months, from a corpus that grew nowhere near that much. The difference is chunks that were re-embedded on edit and never removed. **The headline: across all three engines, 89.73% of what's still live in the index fails at least one check - stale, orphaned or a duplicate**. All three engines agree almost exactly with each other, because staleness and orphan status are pure ledger facts, not engine-dependent behavior. That figure is a real set union (`chunk_id` set operations against the ledger, not a sum of the rows below. Those three categories overlap heavily, so adding their individual percentages would double-count and isn't how this number was computed): **116,798 live chunks are stale.** It's 69.96% of the whole live index, or 76.59% of the 152,504 chunks whose source document still exists. Their source document has since been edited and the index still serves the old version. Identical in pgvector, Qdrant and Chroma.**131,241 chunks (78.61%) are orphaned**- their source doc is deleted, renamed or superseded by a newer version that was never cleaned up. Breakdown: 4,723 from outright-deleted docs, 9,720 from renames and 116,798 from ordinary "the doc was edited and the old chunks never got removed" (failure mode A, the dominant driver).**Every stale chunk is also an orphan**- staleness's predicate is literally the "superseded_live_doc" cause of orphan status, not an independent category, so these two rows are not additive with each other.**~76% of live chunks are duplicates (exact plus near-duplicate combined).** 111,758 (66.9%) are exact byte-identical re-embeds of already-indexed text - this is*not*just the deliberate mid-timeline reingest event. The bigger driver is mundane: editing one paragraph in a multi-chunk doc reprocesses the whole doc into new`chunk_ids` , so every*unchanged*chunk becomes a byte-identical duplicate of itself once re-upserted. If that update landed in the update-without-delete bucket, both copies persist forever. A further cosine-ANN pass (>=0.98 similarity, no ledger ground truth) on the 18,428-chunk remainder (166,947 live chunks minus 111,758 excess exact duplicates minus 36,761 kept canonical originals of those duplicate clusters) found 15,303–15,740 more near-duplicates, engine-dependent (real ANN variance, not a bug). Unlike stale/orphaned, the duplicate set is mostly*not*covered by the other two checks, which is why the true union (89.73%, or 91.30–91.33% including the cosine-ANN layer) comes out*higher*than any individual row above, not lower. Full per-engine union, 7-region breakdown, and the complement (chunks passing every check: 17,147/10.27% conservative, 14,468–14,529/8.67–8.70% including cosine-ANN) are committed inas`results/t2_audit.json` `chunk_set_union_analysis` . **The good news, and it's a real finding, not an absence of testing: deletion itself held up.** I queried for the 637 real git-deleted docs and the 40 GDPR-erased synthetic employees using their own original text as the worst-case query - the query most likely to surface a leak - at both top-5 and top-10, in all three engines. **Zero leaked.** I also checked storage-layer persistence directly (not just "does the API return it") - also zero. For pgvector specifically, the storage layer was dirty first - 254,060 dead tuples sat on disk right after the churn replay and got reclaimed by PostgreSQL's own background autovacuum (no manual VACUUM, ever) before this audit read the heap. The persistence window was real; it just closed on its own. This distinction matters: [EDPB Guidelines 05/2019](https://www.edpb.europa.eu/documents/guideline/guidelines-52019-on-the-criteria-of-the-right-to-be-forgotten-in-the-search_en) are explicit that erasure has to be *verifiable and irreversible* - "the API stopped returning it" alone doesn't satisfy that and you have to check the storage layer too, which is what "Ghost Vectors" ([https://arxiv.org/abs/2606.18497](https://arxiv.org/abs/2606.18497)) found soft-deleted HNSW vectors failing on elsewhere. Here, both checks came back clean, in all three engines. I don't want to overclaim a clean result any more than I'd want to overclaim a dirty one, so I cross-validated the detection mechanism itself. I ran the identical query-and-check pipeline against 200 chunks per engine that are *known to still be live*, confirming each finds itself. Pass rate: 97.5% (pgvector), 98.0% (Qdrant), 98.5% (Chroma) - the small residual gap is ordinary approximate-HNSW recall loss under untuned defaults, isolated by excluding chunks with exact-duplicate siblings (duplicate-tie crowding was initially misleading a naive version of this check down to 86%). If the detection pipeline can find a chunk that's definitely there 97–98% of the time, "zero found" for erased content is a real signal, not a broken probe. ## Ground-truth verification Every check above is scored against the ledger, not just eyeballed: **Staleness detection: 1.0 precision / 1.0 recall** against the independently-re-derived ledger predicate.**Orphan detection: 1.0 precision / 1.0 recall.****Exact-hash duplicate detection: 1.0 precision / 1.0 recall**(true by construction - hash equality is unambiguous). The cosine-ANN layer has no ledger ground truth for "near but not identical" content, so it's reported as an upper bound, not a confirmed count. The methodology behind this post is the same one the paid audit runs on a customer's real index. "Trust me" wouldn't be good enough there, so it doesn't get a pass here either. ## Honest limitations **One corpus.** This is the GitLab Handbook's specific edit patterns - a different corpus with different churn shape could look better or worse.**Defaults, not tuned, and I'm saying so.** All three engines ran their documented default deletion/index behavior. That's deliberate - most teams run defaults - but it means these are not the best numbers each engine could produce with tuning.**Not a speed benchmark.** No cross-engine timing claims anywhere in this post or the raw results.**Synthetic PII is clearly labeled.** It was generated with a fixed Faker seed, kept in its own namespace, never mixed with real handbook attribution.**HNSW build-order nondeterminism** is mitigated (chunk_id-sorted insertion, fixed seed) but not eliminated - rerun variance on the order of single-digit vectors is possible.**Qdrant's server-side deleted-vector count isn't directly measurable** in this server version - I'm reporting the optimizer settings that gate physical reclaim (`deleted_threshold=0.2` ,`vacuum_min_vector_number=1000` ) rather than inventing a number that doesn't exist.**This is a self-built experiment, not a client engagement.** No customer data, no client's actual index.**The check code isn't published, and neither are chunk-level finding IDs.** The corpus, churn replay, ledger schema and raw results are public; the algorithms that turned index state into findings are the paid product.`t2_audit.json` publishes counts and percentages, not the 116,798 individual`chunk_id` values behind the staleness row - so you can re-derive my totals from the ledger, but you can't inspect finding #4,712 without rerunning the pipeline.**I sell an audit built on this exact methodology, so I have a stake in these numbers being credible**- same reason Supabase disclosed their Postgres bias in the pgvector-vs-Qdrant thread. It's also why every number here traces to a published ledger and a committed JSON file rather than to my word. ## Reproduce it Please go to the [experiment repo](https://github.com/rimironenko/rag-index-decay). Fair warning on cost: `corpus.ingest_baseline` embeds ~33k chunks on CPU before `churn.replay` walks the 4,742 commits, so a full independent run is hours, not minutes. Both steps are resumable, and `--pilot 50` / `--max-commits 100` give you a smoke test in a few minutes if you just want to see the machinery work. ## Why not just use X The obvious question. Four adjacent categories exist, and none of them does this. ### RAG eval tools Ragas, Arize Phoenix, LangSmith, DeepEval, TruLens, Future AGI score answer quality (faithfulness, context precision, groundedness) against a sampled question set. Their own vendors' framing is that these metrics assume the retrieval index is trustworthy. That assumption is exactly what breaks here: a pipeline can score 0.95 faithfulness while serving a 13-month-old chunk, because the eval was built against the same stale corpus. Evals sample the questions you thought to ask; this is a census of what's actually in there. Phoenix's embedding-cluster view gets closest to index introspection, but it's a debugging aid, not an inventory. ### Embedding PII scanners [IronCore VectorLens](https://ironcorelabs.com/), BigID answer "is there sensitive data in these vectors?" - a different axis from "should this vector still be here at all?". VectorLens is the closest structural analogue to what I do: self-hosted, read-only, CLI, scans your own exported embeddings. It has no staleness, orphan or duplicate detection, so it's complementary rather than competing. ### Deletion-verification libraries [mnemo](https://github.com/DanceNitra/mnemo) overlap the erasure check specifically, and genuinely so: as of mnemo's v1.9.9 release (July 17, 2026), mnemo's ErasureAuditor re-attempts recovery of an erased subject across registered stores using verbatim scans and NN-inversion, with soft-delete residue probes naming pgvector, Qdrant and Chroma by name. The difference is scope and wiring - it verifies deletion propagation for a *named subject* across stores you register in your application, which is a real and useful thing to have in-app. It doesn't census an external index for staleness, orphans or duplicates, and it needs adapters written into your codebase. ### The vector databases themselves The engines are the subject of the audit, not the auditor. All of them soft-delete by design: pgvector accumulates dead tuples until VACUUM, Qdrant holds tombstones until a segment crosses `deleted_threshold` , Chroma's HNSW grows unbounded. None of them will tell you that a deleted source document is still in your **top-k** - which, to be clear, is precisely the check that came back clean here. ...and the one that isn't a product at all: ### "Just write a SQL cron job comparing timestamps." That is genuinely the mechanism, and for teams with a disciplined document_id→chunk→vector mapping, several of these checks do reduce to a query. The part that doesn't reduce is knowing whether the cron caught everything - which is what the ledger reconciliation above is for, and why the precision/recall numbers are in this post at all. ## If you want this on your own index `rag-staleness-check` The checks that don't need a ledger - staleness (given your own last-modified metadata), orphans (given a source manifest), duplicates and a basic deleted-but-retrievable probe against ids you already know were deleted - are shipping as an open-source, read-only CLI, [ rag-staleness-check](https://github.com/rimironenko/rag-staleness-check) (Apache-2.0). `The paid audit` The multi-engine version, the GDPR Article-17 reporting pack and the ledger-verified precision/recall harness that produced the numbers above are the paid audit that I run as an individual consultant - reach out if you want it run against your real index at [ https://cal.com/rost-rostcamp-y74css/ragproof-discovery](https://cal.com/rost-rostcamp-y74css/ragproof-discovery).