{"slug": "chunked-a-document-and-found-a-new-retrieval-problem", "title": "Chunked a Document and Found a New Retrieval Problem", "summary": "Chunking a long document into 17 overlapping pieces before embedding introduced a retrieval bias: a document split into 17 chunks now has 17 separate entries competing for top-N results, giving it more chances to appear than a short document with one entry. The author, running a local embeddings pipeline with Chroma and Ollama's nomic-embed-text model, also hit two snags: reconnecting to the collection from a different working directory created an empty new database instead of the original 3 entries, and passing raw text via query_texts triggered Chroma's default embedding model, causing a dimension mismatch (384 vs 768). After fixing these by using the original directory and embedding queries with the same model, the collection held 20 entries and queries returned results with distances.", "body_md": "**Context:** Chunking means splitting a long document into smaller overlapping pieces before embedding each one separately, instead of embedding the whole thing (or truncating it, as [Entry 05](https://pipelineandprompts.com/posts/05-local-embeddings-pipeline/) did). This fixes content loss — nothing gets silently dropped — but it introduces a structural question that’s easy to miss: a document split into 17 chunks now has 17 separate entries competing for a spot in search results, while a short document still only has one. More chunks means more chances to appear in a top-N result, independent of whether that chunk is actually the most relevant thing stored.\n\n**Ran:** Chunked [Managed vs Self-Hosted](https://pipelineandprompts.com/posts/managed-vs-self-hosted-handing-over-keys/) (`managed-vs-self-hosted-handing-over-keys.md`, split into 17 pieces at 1000 characters with 200-character overlap) and embedded each chunk into the same Chroma collection from Entries [05](https://pipelineandprompts.com/posts/05-local-embeddings-pipeline/)/[06](https://pipelineandprompts.com/posts/06-querying-embeddings-store/). Two real snags on the way:\n\nFirst, reconnecting to the collection returned an empty database with only the new chunks in it — no sign of the original 3 entries from Entries 05/06. Turned out `PersistentClient(path=\"./chroma_db\")` uses a path relative to wherever Python was launched from, and this session started in a different folder than the earlier ones. A `find` across the filesystem turned up three separate `chroma_db` folders — the “empty” one was actually a brand-new database created by accident, not data loss. Fixed by returning to the original working directory before reconnecting.\n\nSecond, after fixing that, an early query attempt using `collection.query(query_texts=[...])` failed with `InvalidArgumentError: Collection expecting embedding with dimension of 768, got 384` — passing raw text instead of a pre-computed embedding makes Chroma fall back to its own default embedding model, which produces a different vector size than `nomic-embed-text`. Same lesson as Entry 06: always embed the query with the same model used for the documents.\n\nWith that sorted, chunked the new document and embedded each piece into the correct, 20-entry collection:\n\n``` python\nimport ollama\n\ntext = open(\"managed-vs-self-hosted-handing-over-keys.md\").read()\n\nchunk_size = 1000\noverlap = 200\nchunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - overlap)]\n\nfor i, chunk in enumerate(chunks):\n    resp = ollama.embeddings(model=\"nomic-embed-text\", prompt=chunk)\n    collection.upsert(\n        ids=[f\"managed-vs-self-hosted-handing-over-keys_chunk{i}\"],\n        embeddings=[resp[\"embedding\"]],\n        documents=[chunk],\n    )\n\ncollection.count()   # → 20 (3 original entries + 17 new chunks)\n```\n\nThen ran three queries against it — the same “oc pod status” and “pizza topping” questions from Entry 06, plus a real question about the new document’s actual topic. Each question has to be embedded with the same model used for the documents before querying — passing raw text via `query_texts` instead triggers Chroma’s own default embedding model, which produces a different vector size and fails outright:\n\n```\nq1_embed = ollama.embeddings(model=\"nomic-embed-text\", prompt=\"how do I check pod status with oc\")\nq1 = collection.query(query_embeddings=[q1_embed[\"embedding\"]], n_results=3)\n\nq2_embed = ollama.embeddings(model=\"nomic-embed-text\", prompt=\"what's the best pizza topping\")\nq2 = collection.query(query_embeddings=[q2_embed[\"embedding\"]], n_results=3)\n\nq3_embed = ollama.embeddings(model=\"nomic-embed-text\", prompt=\"What are my options for kubernetes, should I use managed or self-hosted Kubernetes\")\nq3 = collection.query(query_embeddings=[q3_embed[\"embedding\"]], n_results=3)\n\nprint(\"Query 1:\", q1[\"ids\"], q1[\"distances\"])\nprint(\"Query 2:\", q2[\"ids\"], q2[\"distances\"])\nprint(\"Query 3:\", q3[\"ids\"], q3[\"distances\"])\n```\n\n**Result:**\n\n| Query | Top 3 matches | Distances | \n|---|---|---|\n| “how do I check pod status with oc” | `02-oc-cli-mentor...` (correct), then 2 unrelated chunks | 437.72, 450.32, 453.22 | \n| “what’s the best pizza topping” | 3 unrelated chunks (all from the new doc) | 519.80, 531.01, 531.51 | \n| “managed or self-hosted Kubernetes” | 3 correct chunks from the new doc | **290.34, 314.06, 316.37** | \n\nTwo things stand out. The on-topic Kubernetes question is the tightest, cleanest match of the whole series so far — every one of the top 3 results came from the right document, at meaningfully lower distances than anything seen in Entries 05 or 06. Chunking clearly works for making a long document’s actual content findable.\n\nBut the oc question shows the tradeoff directly: in Entry 06, its #2 result was the genuinely-related URL entry at distance 499.63. Here, that same document got pushed entirely out of the top 3, replaced by two irrelevant chunks from the 17-chunk document at 450.32 and 453.22 — lower distances not because they’re more relevant, but because a 17-chunk document simply has more entries competing for the middle-ranked spots.\n\n**Takeaway:** Chunking is a real fix for the content-loss problem from Entry 05, and the on-topic result here is the strongest retrieval this series has produced. But it’s not a free upgrade — a document with many chunks crowds out equally-relevant single-entry documents just by having more shots at ranking. Production RAG systems typically handle this with per-document result caps or a re-ranking step after initial retrieval; that’s the natural next thing to test, rather than assuming more chunks always means better search.", "url": "https://wpnews.pro/news/chunked-a-document-and-found-a-new-retrieval-problem", "canonical_source": "https://pipelineandprompts.com/posts/07-chunking-retrieval-bias/", "published_at": "2026-08-27 00:00:00+00:00", "updated_at": "2026-09-07 17:30:27.715064+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "ai-tools"], "entities": ["Chroma", "Ollama", "nomic-embed-text"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/chunked-a-document-and-found-a-new-retrieval-problem", "markdown": "https://wpnews.pro/news/chunked-a-document-and-found-a-new-retrieval-problem.md", "text": "https://wpnews.pro/news/chunked-a-document-and-found-a-new-retrieval-problem.txt", "jsonld": "https://wpnews.pro/news/chunked-a-document-and-found-a-new-retrieval-problem.jsonld"}}