{"slug": "silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index", "title": "Silent Drift: Why Re-Embedding Only on Count Changes Rots Your Semantic Index", "summary": "A developer discovered that their semantic index was returning stale results because the re-embedding job only triggered on file count changes, not content edits. This created 'zombie vectors'—embeddings that no longer matched the source text—and orphaned vectors from deleted or moved files. The developer warns that count-based triggers silently rot an index over time.", "body_md": "My index reported 354 points. The collection had 451. Both numbers were \"correct,\" and that gap is the whole problem.\n\nHere's the setup. I have a wiki that feeds a semantic index. An agent queries that index to pull context before answering infrastructure questions. The re-embedding job was wired to a dead-simple trigger: if the file count changed, re-embed the new files. Cheap, fast, and wrong in a way that takes weeks to notice.\n\nSearch started returning confidently outdated answers. I'd edit a wiki article, fix a stale command, correct a version number, and the agent would keep retrieving the old text as if the edit never happened. No error. No stack trace. The vector was still \"active\" in the database, still matching queries, still ranking high. It just no longer represented what the source file actually said.\n\nThat is the failure mode I've started calling a zombie vector. Technically alive in the index. Semantically dead. It points at a version of the document that stopped existing the moment I saved the edit.\n\nI expected the index to reflect the source. That's the entire contract of a semantic index: the vectors are a searchable projection of the underlying text, and when the text changes, the projection changes with it. I assumed my re-embedding job upheld that contract. It did not. It upheld a much weaker one: the index reflects the *set of files that exist*, not their *contents*.\n\nThose two contracts look identical right up until someone edits a file without adding or removing one. Which, if you maintain a wiki or a memory store, is most of what you do all day.\n\nThe trigger logic was count-based. Pseudocode, but this is close to what it was:\n\n``` python\ndef maybe_reindex(source_dir, index):\n    current_count = count_files(source_dir)\n    indexed_count = index.get_metadata(\"file_count\")\n\n    if current_count == indexed_count:\n        return  # \"nothing changed\", skip\n\n    embed_new_files(source_dir, index)\n    index.set_metadata(\"file_count\", current_count)\n```\n\nRead that `if current_count == indexed_count: return`\n\nline again. That's the bug wearing a helmet. The moment the counts match, the function decides nothing changed and walks away. But file count is invariant under edits. You can rewrite every word of every article, and as long as you don't add or delete a file, the count stays put and the re-embed never fires.\n\nWorse, look at what happens when the count *does* change. `embed_new_files`\n\nonly touches files it considers new. It never revisits existing vectors. So even a triggered run leaves edited-but-not-added files untouched. The index accumulates drift on every edit and only ever grows, never corrects.\n\nI found the 354-vs-451 discrepancy while debugging something unrelated. The metadata field said 354 points because that's the number the job had last written. The collection actually held 451 active points. The extra 97 came from a period where files got split and merged, chunk boundaries moved, and the job appended new chunks without ever retiring the old ones. So now I had two problems stacked on top of each other: vectors that were stale (edited source, unchanged embedding) and vectors that were orphaned (source chunk no longer exists, embedding lingers).\n\nThe orphaned ones are their own special misery. This is the ghost reference problem. A memory file references a path, the path gets deleted or moved, but the embedded chunk still says \"see `scripts/server.py`\n\nfor the handler.\" The agent retrieves that, follows the reference, and hits nothing. The index became a graveyard of pointers to resources that moved on without telling it.\n\nNone of this crashes. That's what makes it rot instead of a bug. A crash you fix in an afternoon. Silent drift you discover months later when someone asks why the agent keeps recommending a flag that got renamed in a release you shipped a while back.\n\nStop asking \"did the count change.\" Start asking \"did the content change.\" The count is a proxy for state, and it's a bad one. The actual state of a file is its contents, and you already have a cheap, exact way to fingerprint contents: a hash.\n\nThe pattern is a manifest. For every source file, store the hash of what you indexed. Before re-embedding, walk the source tree, hash each file, and compare against the manifest. Re-embed only the files whose hashes don't match, plus handle files that vanished.\n\n``` python\nimport hashlib\nimport json\nfrom pathlib import Path\n\ndef file_hash(path: Path) -> str:\n    h = hashlib.sha256()\n    h.update(path.read_bytes())\n    return h.hexdigest()\n\ndef plan_reindex(source_dir: Path, manifest: dict[str, str]):\n    current = {\n        str(p): file_hash(p)\n        for p in source_dir.rglob(\"*.md\")\n    }\n\n    changed = [p for p, h in current.items() if manifest.get(p) != h]\n    deleted = [p for p in manifest if p not in current]\n\n    return changed, deleted, current\n```\n\n`changed`\n\ncatches both brand-new files (no manifest entry, so the `!=`\n\nis true) and edited files (entry exists but the hash moved). `deleted`\n\ncatches files that disappeared, so you can purge their vectors instead of leaving zombies. And the returned `current`\n\ndict becomes your new manifest once the run succeeds.\n\nThe upsert side needs to be idempotent and it needs to delete before it inserts for changed files, otherwise you re-create the 354-vs-451 split. Use a deterministic point ID derived from file path plus chunk index so a re-embed overwrites the same slots rather than appending new ones:\n\n``` php\ndef point_id(file_path: str, chunk_idx: int) -> str:\n    raw = f\"{file_path}::{chunk_idx}\"\n    return hashlib.sha256(raw.encode()).hexdigest()\n\ndef reindex(source_dir, index, manifest):\n    changed, deleted, current = plan_reindex(source_dir, manifest)\n\n    for path in changed + deleted:\n        # remove all chunks for this file first, deterministic IDs\n        index.delete(filter={\"file_path\": path})\n\n    for path in changed:\n        for i, chunk in enumerate(chunk_file(Path(path))):\n            vec = embed(chunk)\n            index.upsert(\n                id=point_id(path, i),\n                vector=vec,\n                payload={\"file_path\": path, \"chunk\": i, \"text\": chunk},\n            )\n\n    save_manifest(current)\n```\n\nDelete-then-insert for changed files means a page that shrank from six chunks to four doesn't leave two ghosts behind. Delete-only for the `deleted`\n\nset purges files that are gone. The manifest save happens last so a crash mid-run leaves you with stale-but-consistent state, and the next run just retries the same diff.\n\nThere's one more guardrail that has nothing to do with counts and everything to do with silent corruption: dimension validation. Embedding providers occasionally return a vector of the wrong length. A model swap, a truncated response, a provider that quietly changed defaults. If you upsert a 768-dim vector into a collection built for 1024, some databases reject it loudly and some accept it and corrupt the distance math. Check before you write:\n\n``` php\nEXPECTED_DIM = 1024\n\ndef embed(text: str) -> list[float]:\n    vec = provider.embed(text)  # e.g. a Bedrock call\n    if len(vec) != EXPECTED_DIM:\n        raise ValueError(\n            f\"embedding dim {len(vec)} != expected {EXPECTED_DIM} \"\n            f\"for text starting {text[:40]!r}\"\n        )\n    return vec\n```\n\nThat single `if len(vec) != EXPECTED_DIM`\n\nhas saved me from a whole category of \"why is search suddenly garbage\" investigations. A dimension mismatch that silently upserts poisons every subsequent query against that collection, and it looks exactly like drift until you check the vector lengths.\n\nThen there's the periodic clean slate. Incremental hash-based diffing is your day-to-day path, but hashing has blind spots. If your chunking logic changes, if your embedding model updates, if the manifest itself gets corrupted, the diff can miss things because the source hash didn't move even though the *right answer* did. So I run a full rebuild on a schedule. Drop the collection, re-embed everything from scratch, write a fresh manifest.\n\n``` python\ndef rebuild_memory_index(source_dir, index):\n    index.recreate_collection(dim=EXPECTED_DIM)\n    manifest = {}\n    for path in source_dir.rglob(\"*.md\"):\n        h = file_hash(path)\n        for i, chunk in enumerate(chunk_file(path)):\n            index.upsert(\n                id=point_id(str(path), i),\n                vector=embed(chunk),\n                payload={\"file_path\": str(path), \"chunk\": i, \"text\": chunk},\n            )\n        manifest[str(path)] = h\n    save_manifest(manifest)\n    print(f\"rebuilt {len(manifest)} files\")\n```\n\nI keep this as `scripts/rebuild-memory-index.py`\n\nand run it the way you'd run a database vacuum. It's the reset button that guarantees the index and source agree, no matter how much drift the incremental path accumulated. My wiki already had a rebuild script; the memory files did not, and that asymmetry was exactly where the documentation drift lived. If one index layer has a rebuild path and another doesn't, the one without it is the one silently rotting.\n\nYou will hit this the moment your source documents outlive their first embedding. Static corpora that never change are fine with count-based logic. But an agent memory store, a wiki, a knowledge base, anything a human or another agent edits in place, drifts continuously. The [six-layer memory architecture I run for Claude Code](https://guatulabs.dev/posts/six-layer-memory-architecture-for-claude-code/) leans hard on the embedding layer being honest, because every layer above it inherits whatever the semantic index believes. A stale vector at the bottom becomes bad reasoning at the top, and the agent has no way to know its retrieved context is a lie.\n\nThe economic argument usually runs the other way. Re-embedding is money and latency, so people optimize to touch only new content. That optimization is real, and for large corpora a full re-index on every write is genuinely wasteful. But the hidden cost sits on the other side of the ledger: an agent reasoning from zombie vectors produces confidently wrong output, and the cost of that, in a system anyone actually depends on, dwarfs the cost of a periodic rebuild. Cheap updates that corrupt the index aren't cheap. They're deferred, and the interest compounds in the form of debugging sessions that start with \"the agent keeps insisting on something that hasn't been true for weeks.\"\n\nThis pairs with a failure mode I've written about before: [your vector DB snapshots landing on the same disk that will fail](https://guatulabs.dev/posts/your-vector-db-snapshots-are-landing-on-the-same-disk-that-will-fail/). Same theme, different layer. In both cases the index looks healthy by the metric you're checking and is quietly broken by the metric that matters. It also sits next to decay-based eviction, which is a *deliberate* forgetting policy; the [ACT-R decay approach](https://guatulabs.dev/posts/eviction-without-deletion-running-an-act-r-decay-policy-for-agent-memory-in-production/) removes vectors on purpose based on activation. Drift is the opposite. It's accidental corruption where the vector stays but its meaning leaves. One is controlled forgetting. The other is uncontrolled lying.\n\nIf you take one thing from this, make it a rule: never trust a count as a proxy for content. Counts answer \"how many things exist,\" and that is almost never the question. The question is \"does the index still agree with the source,\" and only a content fingerprint answers it. Treat re-indexing the way you treat CI. Hash-validate on every change, rebuild from clean state on a schedule, and validate vector dimensions before every write. I've been building this kind of self-correcting index behavior into [agent infrastructure work](https://guatulabs.com/services) because the alternative is an index that reports 354, holds 451, and confidently tells your agent about a file you deleted last month.\n\nThe index that lies to you never throws an error. That's exactly why you have to make it prove it's telling the truth.", "url": "https://wpnews.pro/news/silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index", "canonical_source": "https://dev.to/futhgar/silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index-3k6h", "published_at": "2026-07-29 20:15:48+00:00", "updated_at": "2026-07-29 21:01:55.954749+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index", "markdown": "https://wpnews.pro/news/silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index.md", "text": "https://wpnews.pro/news/silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index.txt", "jsonld": "https://wpnews.pro/news/silent-drift-why-re-embedding-only-on-count-changes-rots-your-semantic-index.jsonld"}}