I let a RAG index rot for 13 months. It grew 5x, 90% failed a check

A 13-month experiment on a real GitLab Handbook corpus found that a RAG index grew 5x from 33,286 to 166,947 chunks, with 90% of chunks failing a ground-truth check due to ingestion bugs. The teardown, conducted by an AWS Community Builder across pgvector, Qdrant, and Chroma, modeled four canonical failure modes including update-without-delete and async deletes, revealing that ingestion errors—not the model—drive index rot.

← ragproof.io / I let a RAG index rot for 13 months. It grew 5x, 90% failed a check. A reproducible teardown of what happens to a vector index after 13 months of real document churn, across pgvector, Qdrant and Chroma. Why I built this I build, consult and teach in the GenAI domain, and I'm an active AWS Community Builder in AI Engineering category. One of my friends came to me with some version of "Why retrieval quality of our in-house RAG degrades over time?" - and I didn't have a satisfying answer other than "let me look". The root cause was removal of a bunch of docs belonging to another track, and this situation inspired me to write this post. I built a version of the worst case on purpose: took a real corpus with real edit history, replayed a year of it the way real ingestion pipelines actually behave including their bugs , and then measured exactly what's wrong with the result. Engine by engine, against a ground-truth ledger that knows what should be true at every point in time. The corpus is the GitLab Handbook https://about.gitlab.com/handbook/ - a real company wiki, thousands of Markdown pages, years of git history, CC BY-SA 4.0 licensed per GitLab's own "Our Handbook is open source: here's why" https://about.gitlab.com/blog/our-handbook-is-open-source-heres-why/ , not the repo's boilerplate LICENSE file, which is silent on content . It reads exactly like the internal wiki or Confluence space most RAG buyers actually index. On top of it: 100 clearly-labeled synthetic employee records Faker, seed=42, not real people for a GDPR erasure demo, since a real company handbook by itself has no personal data to erase. Method T0 → HEAD: picked a commit ~13 months before today d9dfa4eb7a , 2025-06-02 as the starting index, then walked 4,742 real first-parent commits forward to HEAD 6a7e263deb , 2026-07-10, the day I started the experiment - actual edits, deletes, and renames from a real repository's history, not simulated churn. One embedding pass, three engines: every chunk is embedded exactly once BGE-M3, fp32, HF revision 5617a9f61b , sentence-transformers pinned to 5.6.0 and the identical vectors are upserted into pgvector HNSW, cosine , Qdrant cosine , and Chroma cosine - all on documented defaults, no engine given the other's tuning. Docker image digests, client library versions, and the master seed 42 are all pinned and published in the repo. The ingestion bugs are on purpose. Production RAG post-mortems consistently blame ingestion and change-detection, not the model, for index rot. I modeled four canonical failure modes on labeled subsets of the real churn: A Update-without-delete - re-embed and upsert on edit, but never delete the old chunks. The single most common real bug, and the dominant driver of everything below. Applied to 48.44% of update events. B Async/eventually-consistent deletes - issue the delete call but don't wait for it to actually take effect before considering the job done. C Full re-ingest duplication - one mid-timeline "someone re-ran the whole pipeline" event that re-upserts everything under new ids. D Unhandled renames - index the new path as a new doc, leave the old path's chunks orphaned.- Plus a GDPR erasure demo : Article-17-style "delete this person" requests for 40 of the 100 synthetic employees, issued via each engine's normal delete API, with no manual VACUUM/optimize run afterward - because that's what real teams do: call the delete API and assume it's handled. Every single mutation - real or injected-bug - is logged to a SQLite ledger with the expected index state per engine. That ledger is published alongside the repo, which is what makes every number below checkable by someone other than me. Findings Before the percentages, the shape of it: the index went from 33,286 chunks at T0 to 166,947 live chunks at HEAD. Five times the size after 13 months, from a corpus that grew nowhere near that much. The difference is chunks that were re-embedded on edit and never removed. The headline: across all three engines, 89.73% of what's still live in the index fails at least one check - stale, orphaned or a duplicate . All three engines agree almost exactly with each other, because staleness and orphan status are pure ledger facts, not engine-dependent behavior. That figure is a real set union chunk id set operations against the ledger, not a sum of the rows below. Those three categories overlap heavily, so adding their individual percentages would double-count and isn't how this number was computed : 116,798 live chunks are stale. It's 69.96% of the whole live index, or 76.59% of the 152,504 chunks whose source document still exists. Their source document has since been edited and the index still serves the old version. Identical in pgvector, Qdrant and Chroma. 131,241 chunks 78.61% are orphaned - their source doc is deleted, renamed or superseded by a newer version that was never cleaned up. Breakdown: 4,723 from outright-deleted docs, 9,720 from renames and 116,798 from ordinary "the doc was edited and the old chunks never got removed" failure mode A, the dominant driver . Every stale chunk is also an orphan - staleness's predicate is literally the "superseded live doc" cause of orphan status, not an independent category, so these two rows are not additive with each other. ~76% of live chunks are duplicates exact plus near-duplicate combined . 111,758 66.9% are exact byte-identical re-embeds of already-indexed text - this is not just the deliberate mid-timeline reingest event. The bigger driver is mundane: editing one paragraph in a multi-chunk doc reprocesses the whole doc into new chunk ids , so every unchanged chunk becomes a byte-identical duplicate of itself once re-upserted. If that update landed in the update-without-delete bucket, both copies persist forever. A further cosine-ANN pass =0.98 similarity, no ledger ground truth on the 18,428-chunk remainder 166,947 live chunks minus 111,758 excess exact duplicates minus 36,761 kept canonical originals of those duplicate clusters found 15,303–15,740 more near-duplicates, engine-dependent real ANN variance, not a bug . Unlike stale/orphaned, the duplicate set is mostly not covered by the other two checks, which is why the true union 89.73%, or 91.30–91.33% including the cosine-ANN layer comes out higher than any individual row above, not lower. Full per-engine union, 7-region breakdown, and the complement chunks passing every check: 17,147/10.27% conservative, 14,468–14,529/8.67–8.70% including cosine-ANN are committed inas results/t2 audit.json chunk set union analysis . The good news, and it's a real finding, not an absence of testing: deletion itself held up. I queried for the 637 real git-deleted docs and the 40 GDPR-erased synthetic employees using their own original text as the worst-case query - the query most likely to surface a leak - at both top-5 and top-10, in all three engines. Zero leaked. I also checked storage-layer persistence directly not just "does the API return it" - also zero. For pgvector specifically, the storage layer was dirty first - 254,060 dead tuples sat on disk right after the churn replay and got reclaimed by PostgreSQL's own background autovacuum no manual VACUUM, ever before this audit read the heap. The persistence window was real; it just closed on its own. This distinction matters: EDPB Guidelines 05/2019 https://www.edpb.europa.eu/documents/guideline/guidelines-52019-on-the-criteria-of-the-right-to-be-forgotten-in-the-search en are explicit that erasure has to be verifiable and irreversible - "the API stopped returning it" alone doesn't satisfy that and you have to check the storage layer too, which is what "Ghost Vectors" https://arxiv.org/abs/2606.18497 https://arxiv.org/abs/2606.18497 found soft-deleted HNSW vectors failing on elsewhere. Here, both checks came back clean, in all three engines. I don't want to overclaim a clean result any more than I'd want to overclaim a dirty one, so I cross-validated the detection mechanism itself. I ran the identical query-and-check pipeline against 200 chunks per engine that are known to still be live , confirming each finds itself. Pass rate: 97.5% pgvector , 98.0% Qdrant , 98.5% Chroma - the small residual gap is ordinary approximate-HNSW recall loss under untuned defaults, isolated by excluding chunks with exact-duplicate siblings duplicate-tie crowding was initially misleading a naive version of this check down to 86% . If the detection pipeline can find a chunk that's definitely there 97–98% of the time, "zero found" for erased content is a real signal, not a broken probe. Ground-truth verification Every check above is scored against the ledger, not just eyeballed: Staleness detection: 1.0 precision / 1.0 recall against the independently-re-derived ledger predicate. Orphan detection: 1.0 precision / 1.0 recall. Exact-hash duplicate detection: 1.0 precision / 1.0 recall true by construction - hash equality is unambiguous . The cosine-ANN layer has no ledger ground truth for "near but not identical" content, so it's reported as an upper bound, not a confirmed count. The methodology behind this post is the same one the paid audit runs on a customer's real index. "Trust me" wouldn't be good enough there, so it doesn't get a pass here either. Honest limitations One corpus. This is the GitLab Handbook's specific edit patterns - a different corpus with different churn shape could look better or worse. Defaults, not tuned, and I'm saying so. All three engines ran their documented default deletion/index behavior. That's deliberate - most teams run defaults - but it means these are not the best numbers each engine could produce with tuning. Not a speed benchmark. No cross-engine timing claims anywhere in this post or the raw results. Synthetic PII is clearly labeled. It was generated with a fixed Faker seed, kept in its own namespace, never mixed with real handbook attribution. HNSW build-order nondeterminism is mitigated chunk id-sorted insertion, fixed seed but not eliminated - rerun variance on the order of single-digit vectors is possible. Qdrant's server-side deleted-vector count isn't directly measurable in this server version - I'm reporting the optimizer settings that gate physical reclaim deleted threshold=0.2 , vacuum min vector number=1000 rather than inventing a number that doesn't exist. This is a self-built experiment, not a client engagement. No customer data, no client's actual index. The check code isn't published, and neither are chunk-level finding IDs. The corpus, churn replay, ledger schema and raw results are public; the algorithms that turned index state into findings are the paid product. t2 audit.json publishes counts and percentages, not the 116,798 individual chunk id values behind the staleness row - so you can re-derive my totals from the ledger, but you can't inspect finding 4,712 without rerunning the pipeline. I sell an audit built on this exact methodology, so I have a stake in these numbers being credible - same reason Supabase disclosed their Postgres bias in the pgvector-vs-Qdrant thread. It's also why every number here traces to a published ledger and a committed JSON file rather than to my word. Reproduce it Please go to the experiment repo https://github.com/rimironenko/rag-index-decay . Fair warning on cost: corpus.ingest baseline embeds ~33k chunks on CPU before churn.replay walks the 4,742 commits, so a full independent run is hours, not minutes. Both steps are resumable, and --pilot 50 / --max-commits 100 give you a smoke test in a few minutes if you just want to see the machinery work. Why not just use X The obvious question. Four adjacent categories exist, and none of them does this. RAG eval tools Ragas, Arize Phoenix, LangSmith, DeepEval, TruLens, Future AGI score answer quality faithfulness, context precision, groundedness against a sampled question set. Their own vendors' framing is that these metrics assume the retrieval index is trustworthy. That assumption is exactly what breaks here: a pipeline can score 0.95 faithfulness while serving a 13-month-old chunk, because the eval was built against the same stale corpus. Evals sample the questions you thought to ask; this is a census of what's actually in there. Phoenix's embedding-cluster view gets closest to index introspection, but it's a debugging aid, not an inventory. Embedding PII scanners IronCore VectorLens https://ironcorelabs.com/ , BigID answer "is there sensitive data in these vectors?" - a different axis from "should this vector still be here at all?". VectorLens is the closest structural analogue to what I do: self-hosted, read-only, CLI, scans your own exported embeddings. It has no staleness, orphan or duplicate detection, so it's complementary rather than competing. Deletion-verification libraries mnemo https://github.com/DanceNitra/mnemo overlap the erasure check specifically, and genuinely so: as of mnemo's v1.9.9 release July 17, 2026 , mnemo's ErasureAuditor re-attempts recovery of an erased subject across registered stores using verbatim scans and NN-inversion, with soft-delete residue probes naming pgvector, Qdrant and Chroma by name. The difference is scope and wiring - it verifies deletion propagation for a named subject across stores you register in your application, which is a real and useful thing to have in-app. It doesn't census an external index for staleness, orphans or duplicates, and it needs adapters written into your codebase. The vector databases themselves The engines are the subject of the audit, not the auditor. All of them soft-delete by design: pgvector accumulates dead tuples until VACUUM, Qdrant holds tombstones until a segment crosses deleted threshold , Chroma's HNSW grows unbounded. None of them will tell you that a deleted source document is still in your top-k - which, to be clear, is precisely the check that came back clean here. ...and the one that isn't a product at all: "Just write a SQL cron job comparing timestamps." That is genuinely the mechanism, and for teams with a disciplined document id→chunk→vector mapping, several of these checks do reduce to a query. The part that doesn't reduce is knowing whether the cron caught everything - which is what the ledger reconciliation above is for, and why the precision/recall numbers are in this post at all. If you want this on your own index rag-staleness-check The checks that don't need a ledger - staleness given your own last-modified metadata , orphans given a source manifest , duplicates and a basic deleted-but-retrievable probe against ids you already know were deleted - are shipping as an open-source, read-only CLI, rag-staleness-check https://github.com/rimironenko/rag-staleness-check Apache-2.0 . The paid audit The multi-engine version, the GDPR Article-17 reporting pack and the ledger-verified precision/recall harness that produced the numbers above are the paid audit that I run as an individual consultant - reach out if you want it run against your real index at https://cal.com/rost-rostcamp-y74css/ragproof-discovery https://cal.com/rost-rostcamp-y74css/ragproof-discovery .