Tested a Per-Document Cap at Real Scale — and Found I Had No Ground Truth

A developer stress-tested a per-document cap on a retrieval-augmented generation index at real scale, expanding the collection from 20 to 741 chunks, and found the cap traded relevant results for variety rather than fixing a bias. The uncapped top ten was dominated by topically closest documents rather than the longest ones, and the cap replaced extra chunks from on-topic articles with chunks from documents that appeared unrelated by title and distance. The developer also noted that near-verbatim wording scored far better than merely related text, with the top hit being the series' own write-up of the exact question asked.

Entry 07 ended on a loose thread: a document split into 17 chunks crowds out equally relevant single-entry documents, just by having more shots at ranking. The usual fix is a per-document cap, so I flagged it and moved on. Now that the collection holds 741 real chunks instead of 20, it was time to find out whether the problem was real at this size, and whether the fix worked. Measured first, before touching anything. Same question as Entries 06 and 15, ten results, no cap: q = ollama.embeddings model="nomic-embed-text", prompt="how do I check pod status with oc" results = collection.query query embeddings= q "embedding" , n results=10 231.1 08-mcp-server-claude-code chunk2 320.0 15-sse-reconnect-distance-instability chunk1 325.4 mcp-server-architecture-platform-engineering-kubernetes chunk12 328.6 13-hand-rolled-agent-loop chunk2 334.6 mcp-server-architecture-platform-engineering-kubernetes chunk19 344.5 13-hand-rolled-agent-loop chunk4 347.5 13-hand-rolled-agent-loop chunk3 350.6 mcp-server-architecture-platform-engineering-kubernetes chunk16 353.3 mcp-server-architecture-platform-engineering-kubernetes chunk13 357.4 08-mcp-server-claude-code chunk1 Four documents fill all ten slots: four chunks from the MCP architecture article, three from Entry 13, two from Entry 08, one from Entry 15. That looks like the bias I expected, until you check which documents are missing. The three longest articles in the collection the retry-logic piece and the Terraform state piece at 38 chunks each, the zero-downtime deploy piece at 37 don't appear at all. The winners are the topically closest documents, not the longest. Several chunks of one on-topic article can all be legitimately relevant, so some of this clustering isn't bias. The top hit also looked too good. 231.1 is 89 points ahead of the runner-up, and when I printed the chunk, it was Entry 08's own quote of Claude Code's answer, including the phrase "checking pod status with oc". The index was retrieving the write-up of this exact question. That's a side effect of embedding the series' own entries into the store, and it fits the lexical-overlap suspicion from Entry 13: near-verbatim wording scores far better than anything merely related. Then the cap. Fetch forty results, keep only the best chunk per source document, take the top ten: raw = collection.query query embeddings= q "embedding" , n results=40 seen, kept = set , for doc id, dist, meta in zip raw "ids" 0 , raw "distances" 0 , raw "metadatas" 0 : if meta "source" not in seen: seen.add meta "source" kept.append dist, doc id if len kept == 10: break The top four were unchanged, since they were already four different documents. Slots five through ten changed completely: | | Uncapped, slots 5-10 | Capped, slots 5-10 | |---|---|---| | What fills them | More chunks of the MCP architecture article, Entry 13 and Entry 08 | One chunk each from the chunking entry, secrets management, RAG runbooks, provider swapping, zero-downtime deploys, prompt versioning | | Distance range | 334.6 to 357.4 | 360.8 to 381.5 | The cap removed extra chunks from articles that really are about Kubernetes diagnostics over MCP and replaced them with chunks from documents that look unrelated, judging by title and by distance. At this scale it traded relevant results for variety. I had also been treating one result as the real test. Entry 02 was the top hit in Entries 06, 07, 13 and 15, and it vanished from the top ten here, capped or not, so I assumed the cap should bring it back. It didn't. Ranking its chunks across the whole collection shows where it went: 52 415.9 02-oc-cli-mentor-system-prompt chunk1 73 423.6 02-oc-cli-mentor-system-prompt chunk2 77 424.1 02-oc-cli-mentor-system-prompt chunk0 137 441.4 02-oc-cli-mentor-system-prompt chunk3 683 555.5 02-oc-cli-mentor-system-prompt chunk4 Its best chunk scores 415.9, lower better than the 437.7 it scored when it ranked first among 45 entries. The document didn't get worse. Fifty other chunks now sit ahead of it, in a crowded middle where roughly forty chunks fall inside a band about 58 points wide. Entry 15 found that rewording one query moved a distance by 54 points, so that whole band could reshuffle under a different phrasing. I haven't tested that, but the numbers point that way. The more important realization is that Entry 02 was never a correct answer. It's about system prompts, not pod-status commands, and Claude Code said so back in Entry 08. This query probably has no right answer anywhere in the collection, which means I was judging a ranking with nothing to check it against. Whether the cap "worked" isn't a question I can answer from this data. So the finding isn't about the cap. The cap changes results, and at this size it costs relevant ones, but I can't call that good or bad without a question where I already know which document should win. The next entry builds that: a small set of questions, each with a known right document, checked by hand. Until then, treat every ranking in this series as a description of what the index returns, not evidence of what it should return.