{"slug": "show-hn-find-stale-orphaned-deleted-but-retrievable-rag-vectors", "title": "Show HN: Find stale, orphaned, deleted-but-retrievable RAG vectors", "summary": "RAGproof released an open-source tool, rag-staleness-check, that performs read-only staleness, orphan, duplicate, and retrievability checks on a single pgvector, Qdrant, or Chroma index. The tool, installable via pip for Python 3.10+, reports findings to stdout and a JSON file, skipping checks with missing inputs rather than showing misleading zeros. It is the free half of RAGproof's paid audit, which cross-validates findings against a git-history-derived ledger across all three engines.", "body_md": "Read-only staleness / orphan / duplicate / retrievability checks for a\nsingle pgvector, Qdrant or Chroma index - the open-source half of the\n[RAGproof](https://ragproof.io) \"decayed RAG index\" teardown\n([full writeup + multi-engine ledger-verified methodology](https://ragproof.io/blog/rag-index-decay/)).\n\nThis tool runs against **your own** already-indexed vector database and\ntells you:\n\n**staleness**- indexed chunks whose source document has since changed (needs a manifest with`last_modified`\n\nper document and the engine itself storing a per-row last-modified value)**orphans**- indexed chunks whose source document no longer exists in your manifest at all** duplicates**- near-identical chunks (cosine similarity ≥ threshold, default 0.98), plus an exact-hash pass if you store a per-chunk content hash**retrievable-after-delete**- a basic probe: for ids you believe are deleted, is the vector still physically retrievable by id (storage-layer persistence), and does it still surface in top-k search results (functional leak)?\n\nIt's **read-only** - it never calls a write/delete/upsert method on your\nengine. See [Read-only guarantee](#read-only-guarantee) below for how\nthat's enforced, and its limits.\n\nThis is the single-engine, ledger-free, self-serve slice. It doesn't include:\n\n- multi-engine orchestration (run it once per engine yourself)\n- the GDPR Article-17 reporting pack (signed evidence, verifiable-erasure report)\n- engine-specific deep checks (pgvector dead-tuple/VACUUM detail, Qdrant optimizer-threshold detail, Chroma on-disk HNSW growth)\n- a ledger-based precision/recall harness - there's no ground truth here, this tool reports what it finds, not how accurate the finding is\n\nThose live in the private, paid [RAGproof](https://ragproof.io) audit,\nwhich cross-validates every finding against a git-history-derived\nground-truth ledger across all three engines simultaneously.\n\n```\npip install rag-staleness-check              # pgvector only\npip install \"rag-staleness-check[qdrant]\"     # + Qdrant\npip install \"rag-staleness-check[chroma]\"     # + Chroma\npip install \"rag-staleness-check[all]\"        # everything\n```\n\nOr, without installing:\n\n```\npipx run rag-staleness-check --engine pgvector --dsn \"$DSN\" --source ./docs_manifest.json\n```\n\nRequires Python 3.10+.\n\n```\nrag-staleness-check \\\n  --engine pgvector \\\n  --dsn \"postgresql://readonly_user:pw@localhost:5432/mydb\" \\\n  --pg-table chunks \\\n  --pg-doc-id-column doc_id \\\n  --pg-last-modified-column last_modified \\\n  --pg-content-hash-column content_sha256 \\\n  --source ./docs_manifest.json \\\n  --deleted-ids ./deleted_ids.json \\\n  --out findings.json\n```\n\nThis prints a scorecard to stdout and writes the full result to `--out`\n\n(default `findings.json`\n\n). If a check is missing a required input - no\n`--source`\n\n, no per-row metadata configured, no `--deleted-ids`\n\n- it\nreports `\"skipped\": true`\n\nwith a reason instead of silently showing a\nmisleadingly clean 0%.\n\n| Engine | `--dsn` format |\nExample |\n|---|---|---|\n`pgvector` |\nPostgres DSN | `postgresql://user:pw@localhost:5432/db` |\n`qdrant` |\nBase URL | `http://localhost:6333` |\n`chroma` |\n`host:port` |\n`localhost:8000` |\n\nYour table/column or payload/metadata field names are your own - this tool doesn't assume anything about your schema beyond what you tell it via flags:\n\n**pgvector** (`--pg-*`\n\n): `--pg-table`\n\n(required), `--pg-id-column`\n\n(default `id`\n\n), `--pg-vector-column`\n\n(default `embedding`\n\n),\n`--pg-doc-id-column`\n\n, `--pg-last-modified-column`\n\n,\n`--pg-content-hash-column`\n\n. The last three are optional, and leaving them\nout is exactly what makes staleness / orphans / the duplicates check's\nexact-hash pass report `skipped`\n\n/ zero-coverage.\n\n**Qdrant / Chroma** (`--collection`\n\nrequired, plus payload/metadata field\nnames): `--doc-id-field`\n\n(default `doc_id`\n\n), `--last-modified-field`\n\n(default `last_modified`\n\n), `--content-hash-field`\n\n(default\n`content_sha256`\n\n).\n\nA JSON file describing what documents *should* exist - this is the join\nkey for staleness and orphans. There's no ledger in this tool; the\nmanifest is the only source of truth about \"what's current.\"\n\n```\n{\n  \"manifest_version\": 1,\n  \"documents\": [\n    {\n      \"doc_id\": \"handbook/engineering/on-call.md\",\n      \"last_modified\": \"2026-06-01T00:00:00Z\",\n      \"content_sha256\": \"optional, doc-level, informational only\"\n    }\n  ]\n}\n```\n\n`doc_id`\n\nhas to match whatever value your own ingestion pipeline wrote\ninto each indexed chunk's doc-id column/field (see schema mapping above).\n`last_modified`\n\nis only needed if you want the staleness check to run.\n`content_sha256`\n\nhere is doc-level and purely informational - it isn't\nthe input to duplicate detection (see below). Full example at\n[ examples/docs_manifest.json](https://github.com/rimironenko/rag-staleness-check/blob/v0.1.1/examples/docs_manifest.json).\n\nA JSON array of ids you believe are deleted:\n\n```\n[\"chunk-abc123\", \"chunk-def456\"]\n```\n\nFor each one, this checks whether the vector is still retrievable by id\n(storage-layer persistence) and, if so, whether it still shows up in a\ntop-k self-query (functional leak). The distinction matters: per\n[EDPB Guidelines 05/2019](https://www.edpb.europa.eu/documents/guideline/guidelines-52019-on-the-criteria-of-the-right-to-be-forgotten-in-the-search_en),\nerasure has to be *verifiable and irreversible* - suppressing a record\nfrom search results isn't enough on its own if the underlying vector is\nstill there. `findings.json`\n\n's `retrievable.framing`\n\nfield spells this\nout every run.\n\nThis tool never calls a write/delete/upsert method on your engine, and that's enforced two ways:\n\n**Structurally, via a static test**-walks the actual syntax tree of every file under`tests/test_no_write_methods.py`\n\n`src/rag_staleness_check/`\n\nand fails the build if any write/delete/ upsert-shaped method gets called, or any SQL write statement is passed to`execute()`\n\n/`executemany()`\n\n. Not a substring grep - that would false-positive on this very README and on docstrings - it parses real syntax.**At runtime, for pgvector**- every connection opens with`SET default_transaction_read_only = on`\n\n, and`rag_staleness_check.readonly.assert_pg_read_only()`\n\nre-checks`SHOW default_transaction_read_only`\n\nbefore any query runs, refusing to proceed if it isn't`'on'`\n\n. That's defense-in-depth against a connection pooler (e.g. PgBouncer) silently swallowing the session-level`SET`\n\n.\n\nQdrant and Chroma don't give a client-exposed way to check \"is this session/API key read-only\" - that's deployment-side RBAC, not something either client library reports. For those two engines, enforcement is the static test above plus your own deployment-side scoping: connect with a read-only-scoped API key/token if your engine supports one. There's no runtime assertion for Qdrant/Chroma equivalent to pgvector's - flagging that here rather than implying parity where there isn't any.\n\nFor extra assurance on pgvector, connect through a locked-down role instead of relying on this tool alone:\n\n```\nCREATE ROLE rag_staleness_check_readonly NOSUPERUSER LOGIN PASSWORD '...';\nGRANT CONNECT ON DATABASE mydb TO rag_staleness_check_readonly;\nGRANT USAGE ON SCHEMA public TO rag_staleness_check_readonly;\nGRANT SELECT ON chunks TO rag_staleness_check_readonly;\n```\n\n**No default telemetry.** Nothing is collected or sent automatically.\n`--share-anonymous-scorecard`\n\nis an explicit opt-in that prints the exact\nanonymized payload (aggregate counts + engine type only - never your dsn,\nhostnames, doc_ids, or chunk_ids) that *would* be shared. There's no\nbackend configured yet, so nothing actually goes over the network - the\nflag is reserved for a future opt-in submission endpoint.\n`DO_NOT_TRACK`\n\nin the environment, if set, forces this off even if the\nflag is passed.\n\nSeparately, this tool disables **chromadb's own client-side telemetry**\n(`anonymized_telemetry=False`\n\n) when connecting to Chroma. The `chromadb`\n\npackage ships its own default-on PostHog-based telemetry, independent of\nanything above - leaving it enabled would quietly break the \"no default\ntelemetry\" promise for `--engine chroma`\n\nusers even though this tool's\nown code never phones home.\n\nOne cosmetic wrinkle: on `chromadb==0.6.3`\n\n, the setting takes effect\n(verified via `client._system.settings.anonymized_telemetry is False`\n\n),\nbut a startup event (`ClientStartEvent`\n\n) still tries to fire through a\ncode path that doesn't consistently honor it, and throws\n`capture() takes 1 positional argument but 3 were given`\n\nwhile\n*constructing* the call - before any request is made. You may see this\nline printed; it doesn't mean anything was sent.\n\n**Duplicate detection's exact-hash pass** needs a per-chunk content hash stored in your engine (`--pg-content-hash-column`\n\n/`--content-hash-field`\n\n). Without one, it falls back to cosine-ANN only, and a`warning`\n\nfield in the`duplicates`\n\ncheck says so rather than staying silent about it.**Chroma's** only reports a live count, intentionally`snapshot_stats()`\n\n- no on-disk HNSW-directory-size measurement. (The private RAGproof\naudit's dev-environment build of this used a\n`docker exec du`\n\ntrick against a known-local container; that has no equivalent against a real third-party deployment, so it isn't shipped here.)\n\n- no on-disk HNSW-directory-size measurement. (The private RAGproof\naudit's dev-environment build of this used a\nmay warn if your installed client's minor version differs from your server's by more than 1 (e.g. \"Qdrant client version 1.18.0 is incompatible with server version 1.15.5\"). Non-fatal - the check still runs - but if it's noisy, pin`qdrant-client`\n\n's own compatibility check`qdrant-client`\n\nto match your server's minor version yourself.**This tool reports counts and pairs, not precision/recall.** There's no ground truth available client-side to score itself against - that's what the private, ledger-based audit is for.**Duplicate detection needs a retrievable vector per candidate**- a chunk with no vector returned by`get_vector`\n\nis silently excluded from the cosine-ANN pass (nothing to query with), not counted as \"not a duplicate.\"\n\n`qdrant-client`\n\nand`chromadb`\n\nare**optional extras**(`pip install rag-staleness-check[qdrant]`\n\n/`[chroma]`\n\n/`[all]`\n\n) so a pgvector-only user doesn't have to pull in Chroma's heavier transitive dependencies.- Neither is pinned to an exact version. This tool connects to a third party's already-running deployment, whose server version is out of its control, so hard-pinning - unlike a project that ships and controls its own Docker images - would just break users on anything else.\n\nApache-2.0. See\n[LICENSE](https://github.com/rimironenko/rag-staleness-check/blob/v0.1.1/LICENSE).\nIndependent of any other RAGproof project's license.\n\nIssues and PRs welcome. Not yet implemented: a `--method minhash`\n\nsurface-text-based duplicate detection mode (via `datasketch`\n\n) as an\nalternative/complement to the cosine-embedding approach above - PRs\nwelcome.", "url": "https://wpnews.pro/news/show-hn-find-stale-orphaned-deleted-but-retrievable-rag-vectors", "canonical_source": "https://github.com/rimironenko/rag-staleness-check", "published_at": "2026-08-06 17:13:54+00:00", "updated_at": "2026-08-12 04:43:01.288683+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "machine-learning"], "entities": ["RAGproof", "pgvector", "Qdrant", "Chroma", "rag-staleness-check"], "alternates": {"html": "https://wpnews.pro/news/show-hn-find-stale-orphaned-deleted-but-retrievable-rag-vectors", "markdown": "https://wpnews.pro/news/show-hn-find-stale-orphaned-deleted-but-retrievable-rag-vectors.md", "text": "https://wpnews.pro/news/show-hn-find-stale-orphaned-deleted-but-retrievable-rag-vectors.txt", "jsonld": "https://wpnews.pro/news/show-hn-find-stale-orphaned-deleted-but-retrievable-rag-vectors.jsonld"}}