# Chunked a Document and Found a New Retrieval Problem

> Source: <https://pipelineandprompts.com/posts/07-chunking-retrieval-bias/>
> Published: 2026-08-27 00:00:00+00:00

**Context:** Chunking means splitting a long document into smaller overlapping pieces before embedding each one separately, instead of embedding the whole thing (or truncating it, as [Entry 05](https://pipelineandprompts.com/posts/05-local-embeddings-pipeline/) did). This fixes content loss — nothing gets silently dropped — but it introduces a structural question that’s easy to miss: a document split into 17 chunks now has 17 separate entries competing for a spot in search results, while a short document still only has one. More chunks means more chances to appear in a top-N result, independent of whether that chunk is actually the most relevant thing stored.

**Ran:** Chunked [Managed vs Self-Hosted](https://pipelineandprompts.com/posts/managed-vs-self-hosted-handing-over-keys/) (`managed-vs-self-hosted-handing-over-keys.md`, split into 17 pieces at 1000 characters with 200-character overlap) and embedded each chunk into the same Chroma collection from Entries [05](https://pipelineandprompts.com/posts/05-local-embeddings-pipeline/)/[06](https://pipelineandprompts.com/posts/06-querying-embeddings-store/). Two real snags on the way:

First, reconnecting to the collection returned an empty database with only the new chunks in it — no sign of the original 3 entries from Entries 05/06. Turned out `PersistentClient(path="./chroma_db")` uses a path relative to wherever Python was launched from, and this session started in a different folder than the earlier ones. A `find` across the filesystem turned up three separate `chroma_db` folders — the “empty” one was actually a brand-new database created by accident, not data loss. Fixed by returning to the original working directory before reconnecting.

Second, after fixing that, an early query attempt using `collection.query(query_texts=[...])` failed with `InvalidArgumentError: Collection expecting embedding with dimension of 768, got 384` — passing raw text instead of a pre-computed embedding makes Chroma fall back to its own default embedding model, which produces a different vector size than `nomic-embed-text`. Same lesson as Entry 06: always embed the query with the same model used for the documents.

With that sorted, chunked the new document and embedded each piece into the correct, 20-entry collection:

``` python
import ollama

text = open("managed-vs-self-hosted-handing-over-keys.md").read()

chunk_size = 1000
overlap = 200
chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - overlap)]

for i, chunk in enumerate(chunks):
    resp = ollama.embeddings(model="nomic-embed-text", prompt=chunk)
    collection.upsert(
        ids=[f"managed-vs-self-hosted-handing-over-keys_chunk{i}"],
        embeddings=[resp["embedding"]],
        documents=[chunk],
    )

collection.count()   # → 20 (3 original entries + 17 new chunks)
```

Then ran three queries against it — the same “oc pod status” and “pizza topping” questions from Entry 06, plus a real question about the new document’s actual topic. Each question has to be embedded with the same model used for the documents before querying — passing raw text via `query_texts` instead triggers Chroma’s own default embedding model, which produces a different vector size and fails outright:

```
q1_embed = ollama.embeddings(model="nomic-embed-text", prompt="how do I check pod status with oc")
q1 = collection.query(query_embeddings=[q1_embed["embedding"]], n_results=3)

q2_embed = ollama.embeddings(model="nomic-embed-text", prompt="what's the best pizza topping")
q2 = collection.query(query_embeddings=[q2_embed["embedding"]], n_results=3)

q3_embed = ollama.embeddings(model="nomic-embed-text", prompt="What are my options for kubernetes, should I use managed or self-hosted Kubernetes")
q3 = collection.query(query_embeddings=[q3_embed["embedding"]], n_results=3)

print("Query 1:", q1["ids"], q1["distances"])
print("Query 2:", q2["ids"], q2["distances"])
print("Query 3:", q3["ids"], q3["distances"])
```

**Result:**

| Query | Top 3 matches | Distances | 
|---|---|---|
| “how do I check pod status with oc” | `02-oc-cli-mentor...` (correct), then 2 unrelated chunks | 437.72, 450.32, 453.22 | 
| “what’s the best pizza topping” | 3 unrelated chunks (all from the new doc) | 519.80, 531.01, 531.51 | 
| “managed or self-hosted Kubernetes” | 3 correct chunks from the new doc | **290.34, 314.06, 316.37** | 

Two things stand out. The on-topic Kubernetes question is the tightest, cleanest match of the whole series so far — every one of the top 3 results came from the right document, at meaningfully lower distances than anything seen in Entries 05 or 06. Chunking clearly works for making a long document’s actual content findable.

But the oc question shows the tradeoff directly: in Entry 06, its #2 result was the genuinely-related URL entry at distance 499.63. Here, that same document got pushed entirely out of the top 3, replaced by two irrelevant chunks from the 17-chunk document at 450.32 and 453.22 — lower distances not because they’re more relevant, but because a 17-chunk document simply has more entries competing for the middle-ranked spots.

**Takeaway:** Chunking is a real fix for the content-loss problem from Entry 05, and the on-topic result here is the strongest retrieval this series has produced. But it’s not a free upgrade — a document with many chunks crowds out equally-relevant single-entry documents just by having more shots at ranking. Production RAG systems typically handle this with per-document result caps or a re-ranking step after initial retrieval; that’s the natural next thing to test, rather than assuming more chunks always means better search.