Chunked a Document and Found a New Retrieval Problem

Context: Chunking means splitting a long document into smaller overlapping pieces before embedding each one separately, instead of embedding the whole thing (or truncating it, as Entry 05 did). This fixes content loss — nothing gets silently dropped — but it introduces a structural question that’s easy to miss: a document split into 17 chunks now has 17 separate entries competing for a spot in search results, while a short document still only has one. More chunks means more chances to appear in a top-N result, independent of whether that chunk is actually the most relevant thing stored.

Ran: Chunked Managed vs Self-Hosted (managed-vs-self-hosted-handing-over-keys.md, split into 17 pieces at 1000 characters with 200-character overlap) and embedded each chunk into the same Chroma collection from Entries 05/06. Two real snags on the way:

First, reconnecting to the collection returned an empty database with only the new chunks in it — no sign of the original 3 entries from Entries 05/06. Turned out PersistentClient(path="./chroma_db") uses a path relative to wherever Python was launched from, and this session started in a different folder than the earlier ones. A find across the filesystem turned up three separate chroma_db folders — the “empty” one was actually a brand-new database created by accident, not data loss. Fixed by returning to the original working directory before reconnecting.

Second, after fixing that, an early query attempt using collection.query(query_texts=[...]) failed with InvalidArgumentError: Collection expecting embedding with dimension of 768, got 384 — passing raw text instead of a pre-computed embedding makes Chroma fall back to its own default embedding model, which produces a different vector size than nomic-embed-text. Same lesson as Entry 06: always embed the query with the same model used for the documents.

With that sorted, chunked the new document and embedded each piece into the correct, 20-entry collection:

import ollama

text = open("managed-vs-self-hosted-handing-over-keys.md").read()

chunk_size = 1000
overlap = 200
chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - overlap)]

for i, chunk in enumerate(chunks):
    resp = ollama.embeddings(model="nomic-embed-text", prompt=chunk)
    collection.upsert(
        ids=[f"managed-vs-self-hosted-handing-over-keys_chunk{i}"],
        embeddings=[resp["embedding"]],
        documents=[chunk],
    )

collection.count()   # → 20 (3 original entries + 17 new chunks)

Then ran three queries against it — the same “oc pod status” and “pizza topping” questions from Entry 06, plus a real question about the new document’s actual topic. Each question has to be embedded with the same model used for the documents before querying — passing raw text via query_texts instead triggers Chroma’s own default embedding model, which produces a different vector size and fails outright:

q1_embed = ollama.embeddings(model="nomic-embed-text", prompt="how do I check pod status with oc")
q1 = collection.query(query_embeddings=[q1_embed["embedding"]], n_results=3)

q2_embed = ollama.embeddings(model="nomic-embed-text", prompt="what's the best pizza topping")
q2 = collection.query(query_embeddings=[q2_embed["embedding"]], n_results=3)

q3_embed = ollama.embeddings(model="nomic-embed-text", prompt="What are my options for kubernetes, should I use managed or self-hosted Kubernetes")
q3 = collection.query(query_embeddings=[q3_embed["embedding"]], n_results=3)

print("Query 1:", q1["ids"], q1["distances"])
print("Query 2:", q2["ids"], q2["distances"])
print("Query 3:", q3["ids"], q3["distances"])

Result:

Two things stand out. The on-topic Kubernetes question is the tightest, cleanest match of the whole series so far — every one of the top 3 results came from the right document, at meaningfully lower distances than anything seen in Entries 05 or 06. Chunking clearly works for making a long document’s actual content findable.

But the oc question shows the tradeoff directly: in Entry 06, its #2 result was the genuinely-related URL entry at distance 499.63. Here, that same document got pushed entirely out of the top 3, replaced by two irrelevant chunks from the 17-chunk document at 450.32 and 453.22 — lower distances not because they’re more relevant, but because a 17-chunk document simply has more entries competing for the middle-ranked spots.

Takeaway: Chunking is a real fix for the content-loss problem from Entry 05, and the on-topic result here is the strongest retrieval this series has produced. But it’s not a free upgrade — a document with many chunks crowds out equally-relevant single-entry documents just by having more shots at ranking. Production RAG systems typically handle this with per-document result caps or a re-ranking step after initial retrieval; that’s the natural next thing to test, rather than assuming more chunks always means better search.