My index reported 354 points. The collection had 451. Both numbers were "correct," and that gap is the whole problem.
Here's the setup. I have a wiki that feeds a semantic index. An agent queries that index to pull context before answering infrastructure questions. The re-embedding job was wired to a dead-simple trigger: if the file count changed, re-embed the new files. Cheap, fast, and wrong in a way that takes weeks to notice.
Search started returning confidently outdated answers. I'd edit a wiki article, fix a stale command, correct a version number, and the agent would keep retrieving the old text as if the edit never happened. No error. No stack trace. The vector was still "active" in the database, still matching queries, still ranking high. It just no longer represented what the source file actually said.
That is the failure mode I've started calling a zombie vector. Technically alive in the index. Semantically dead. It points at a version of the document that stopped existing the moment I saved the edit.
I expected the index to reflect the source. That's the entire contract of a semantic index: the vectors are a searchable projection of the underlying text, and when the text changes, the projection changes with it. I assumed my re-embedding job upheld that contract. It did not. It upheld a much weaker one: the index reflects the set of files that exist, not their contents.
Those two contracts look identical right up until someone edits a file without adding or removing one. Which, if you maintain a wiki or a memory store, is most of what you do all day.
The trigger logic was count-based. Pseudocode, but this is close to what it was:
def maybe_reindex(source_dir, index):
current_count = count_files(source_dir)
indexed_count = index.get_metadata("file_count")
if current_count == indexed_count:
return # "nothing changed", skip
embed_new_files(source_dir, index)
index.set_metadata("file_count", current_count)
Read that if current_count == indexed_count: return
line again. That's the bug wearing a helmet. The moment the counts match, the function decides nothing changed and walks away. But file count is invariant under edits. You can rewrite every word of every article, and as long as you don't add or delete a file, the count stays put and the re-embed never fires.
Worse, look at what happens when the count does change. embed_new_files
only touches files it considers new. It never revisits existing vectors. So even a triggered run leaves edited-but-not-added files untouched. The index accumulates drift on every edit and only ever grows, never corrects.
I found the 354-vs-451 discrepancy while debugging something unrelated. The metadata field said 354 points because that's the number the job had last written. The collection actually held 451 active points. The extra 97 came from a period where files got split and merged, chunk boundaries moved, and the job appended new chunks without ever retiring the old ones. So now I had two problems stacked on top of each other: vectors that were stale (edited source, unchanged embedding) and vectors that were orphaned (source chunk no longer exists, embedding lingers).
The orphaned ones are their own special misery. This is the ghost reference problem. A memory file references a path, the path gets deleted or moved, but the embedded chunk still says "see scripts/server.py
for the handler." The agent retrieves that, follows the reference, and hits nothing. The index became a graveyard of pointers to resources that moved on without telling it.
None of this crashes. That's what makes it rot instead of a bug. A crash you fix in an afternoon. Silent drift you discover months later when someone asks why the agent keeps recommending a flag that got renamed in a release you shipped a while back.
Stop asking "did the count change." Start asking "did the content change." The count is a proxy for state, and it's a bad one. The actual state of a file is its contents, and you already have a cheap, exact way to fingerprint contents: a hash.
The pattern is a manifest. For every source file, store the hash of what you indexed. Before re-embedding, walk the source tree, hash each file, and compare against the manifest. Re-embed only the files whose hashes don't match, plus handle files that vanished.
import hashlib
import json
from pathlib import Path
def file_hash(path: Path) -> str:
h = hashlib.sha256()
h.update(path.read_bytes())
return h.hexdigest()
def plan_reindex(source_dir: Path, manifest: dict[str, str]):
current = {
str(p): file_hash(p)
for p in source_dir.rglob("*.md")
}
changed = [p for p, h in current.items() if manifest.get(p) != h]
deleted = [p for p in manifest if p not in current]
return changed, deleted, current
changed
catches both brand-new files (no manifest entry, so the !=
is true) and edited files (entry exists but the hash moved). deleted
catches files that disappeared, so you can purge their vectors instead of leaving zombies. And the returned current
dict becomes your new manifest once the run succeeds.
The upsert side needs to be idempotent and it needs to delete before it inserts for changed files, otherwise you re-create the 354-vs-451 split. Use a deterministic point ID derived from file path plus chunk index so a re-embed overwrites the same slots rather than appending new ones:
def point_id(file_path: str, chunk_idx: int) -> str:
raw = f"{file_path}::{chunk_idx}"
return hashlib.sha256(raw.encode()).hexdigest()
def reindex(source_dir, index, manifest):
changed, deleted, current = plan_reindex(source_dir, manifest)
for path in changed + deleted:
index.delete(filter={"file_path": path})
for path in changed:
for i, chunk in enumerate(chunk_file(Path(path))):
vec = embed(chunk)
index.upsert(
id=point_id(path, i),
vector=vec,
payload={"file_path": path, "chunk": i, "text": chunk},
)
save_manifest(current)
Delete-then-insert for changed files means a page that shrank from six chunks to four doesn't leave two ghosts behind. Delete-only for the deleted
set purges files that are gone. The manifest save happens last so a crash mid-run leaves you with stale-but-consistent state, and the next run just retries the same diff.
There's one more guardrail that has nothing to do with counts and everything to do with silent corruption: dimension validation. Embedding providers occasionally return a vector of the wrong length. A model swap, a truncated response, a provider that quietly changed defaults. If you upsert a 768-dim vector into a collection built for 1024, some databases reject it loudly and some accept it and corrupt the distance math. Check before you write:
EXPECTED_DIM = 1024
def embed(text: str) -> list[float]:
vec = provider.embed(text) # e.g. a Bedrock call
if len(vec) != EXPECTED_DIM:
raise ValueError(
f"embedding dim {len(vec)} != expected {EXPECTED_DIM} "
f"for text starting {text[:40]!r}"
)
return vec
That single if len(vec) != EXPECTED_DIM
has saved me from a whole category of "why is search suddenly garbage" investigations. A dimension mismatch that silently upserts poisons every subsequent query against that collection, and it looks exactly like drift until you check the vector lengths.
Then there's the periodic clean slate. Incremental hash-based diffing is your day-to-day path, but hashing has blind spots. If your chunking logic changes, if your embedding model updates, if the manifest itself gets corrupted, the diff can miss things because the source hash didn't move even though the right answer did. So I run a full rebuild on a schedule. Drop the collection, re-embed everything from scratch, write a fresh manifest.
def rebuild_memory_index(source_dir, index):
index.recreate_collection(dim=EXPECTED_DIM)
manifest = {}
for path in source_dir.rglob("*.md"):
h = file_hash(path)
for i, chunk in enumerate(chunk_file(path)):
index.upsert(
id=point_id(str(path), i),
vector=embed(chunk),
payload={"file_path": str(path), "chunk": i, "text": chunk},
)
manifest[str(path)] = h
save_manifest(manifest)
print(f"rebuilt {len(manifest)} files")
I keep this as scripts/rebuild-memory-index.py
and run it the way you'd run a database vacuum. It's the reset button that guarantees the index and source agree, no matter how much drift the incremental path accumulated. My wiki already had a rebuild script; the memory files did not, and that asymmetry was exactly where the documentation drift lived. If one index layer has a rebuild path and another doesn't, the one without it is the one silently rotting.
You will hit this the moment your source documents outlive their first embedding. Static corpora that never change are fine with count-based logic. But an agent memory store, a wiki, a knowledge base, anything a human or another agent edits in place, drifts continuously. The six-layer memory architecture I run for Claude Code leans hard on the embedding layer being honest, because every layer above it inherits whatever the semantic index believes. A stale vector at the bottom becomes bad reasoning at the top, and the agent has no way to know its retrieved context is a lie.
The economic argument usually runs the other way. Re-embedding is money and latency, so people optimize to touch only new content. That optimization is real, and for large corpora a full re-index on every write is genuinely wasteful. But the hidden cost sits on the other side of the ledger: an agent reasoning from zombie vectors produces confidently wrong output, and the cost of that, in a system anyone actually depends on, dwarfs the cost of a periodic rebuild. Cheap updates that corrupt the index aren't cheap. They're deferred, and the interest compounds in the form of debugging sessions that start with "the agent keeps insisting on something that hasn't been true for weeks."
This pairs with a failure mode I've written about before: your vector DB snapshots landing on the same disk that will fail. Same theme, different layer. In both cases the index looks healthy by the metric you're checking and is quietly broken by the metric that matters. It also sits next to decay-based eviction, which is a deliberate forgetting policy; the ACT-R decay approach removes vectors on purpose based on activation. Drift is the opposite. It's accidental corruption where the vector stays but its meaning leaves. One is controlled forgetting. The other is uncontrolled lying.
If you take one thing from this, make it a rule: never trust a count as a proxy for content. Counts answer "how many things exist," and that is almost never the question. The question is "does the index still agree with the source," and only a content fingerprint answers it. Treat re-indexing the way you treat CI. Hash-validate on every change, rebuild from clean state on a schedule, and validate vector dimensions before every write. I've been building this kind of self-correcting index behavior into agent infrastructure work because the alternative is an index that reports 354, holds 451, and confidently tells your agent about a file you deleted last month.
The index that lies to you never throws an error. That's exactly why you have to make it prove it's telling the truth.