{"slug": "documents-aren-t-bags-of-chunks", "title": "Documents Aren't Bags of Chunks", "summary": "A developer argues that fixed-size chunking in RAG pipelines destroys document structure, which carries meaning. Instead of asking where to split a document, the developer suggests asking what the smallest unit is that still makes sense on its own, such as logical boundaries like sections or code blocks with explanations. The post contends that retrieval failures often stem from dismantling documents before indexing, not from the retrieval itself.", "body_md": "Imagine taking a technical specification and cutting it with scissors every five hundred words.\n\nNo attention to headings.\n\nNo respect for sections.\n\nNo care for where a thought begins and ends.\n\nJust cut.\n\nStack.\n\nIndex.\n\nThat sounds absurd.\n\nYet this is remarkably close to what many retrieval pipelines do before a document ever reaches a vector database.\n\nOne of the most common discussions around RAG sounds like this:\n\nWhat's the optimal chunk size?\n\n256 tokens?\n\n512?\n\nShould chunks overlap?\n\nAfter watching enough retrieval failures, I started thinking this isn't the right question.\n\nThe real question isn't **where** to split a document.\n\nIt's **whether every document should be split the same way at all.**\n\nSuppose you have a technical specification called **\"Rate Limiting Strategy.\"**\n\nIt contains four sections:\n\nNow index it using fixed-size chunks.\n\nThe heading lands in one chunk.\n\nThe code example ends up in another.\n\nThe explanation of edge cases becomes a third.\n\nEach chunk receives its own embedding.\n\nEach becomes an independent point in vector space.\n\nThe text survives.\n\nThe document, as a coherent unit of meaning, does not.\n\nNow someone asks:\n\n\"What happens when a user hits the rate limit?\"\n\nThe retriever returns the chunk containing the code.\n\nFrom its perspective, everything worked.\n\nThe identifiers match.\n\nThe embeddings are close.\n\nThe similarity score is high.\n\nBut something important has disappeared.\n\nThe heading explaining *what* this code belongs to is somewhere else.\n\nThe paragraph explaining *why* the algorithm behaves this way is somewhere else.\n\nThe exceptions are somewhere else.\n\nThe model receives valid code—but stripped of the reasoning that made the code meaningful.\n\nIt can explain **what** the function does.\n\nIt cannot explain **why** the system behaves that way.\n\nIt's easy to think of document structure as presentation.\n\nI don't.\n\nA heading tells the reader:\n\nEverything below belongs to the same idea.\n\nA section boundary says:\n\nThis thought ends here.\n\nA list says:\n\nThese items should be understood together.\n\nThose aren't visual decorations.\n\nThey're part of the information itself.\n\nWhen those signals disappear during indexing, retrieval loses something that humans almost never notice consciously—but rely on constantly while reading.\n\nEventually I stopped asking:\n\nHow should I split this document?\n\nInstead I started asking:\n\nWhat is the smallest unit that still makes sense on its own?\n\nSometimes that's a paragraph.\n\nSometimes it's an entire section.\n\nSometimes it's a code block together with the explanation above it.\n\nSometimes it's a diagram and its caption.\n\nAn architecture decision record often makes sense only when the decision, its rationale, and its consequences stay together.\n\nAn API reference is easier to understand when an endpoint remains connected to its parameters and examples.\n\nMeeting notes become much more useful when decisions stay attached to owners and action items.\n\nThe right boundary isn't a character count.\n\nIt's a logical boundary.\n\nOne pattern kept repeating itself.\n\nThe retrieval engine successfully found text related to the question.\n\nThe model confidently explained that text.\n\nAnd yet the actual answer was still missing.\n\nNot because retrieval failed.\n\nBecause the document had already been dismantled before retrieval even began.\n\nI've gradually stopped thinking of documents as long strings of text.\n\nThey're closer to arguments.\n\nSomeone deliberately grouped ideas together.\n\nSeparated topics.\n\nAdded headings.\n\nBuilt a flow of reasoning.\n\nThat structure carries meaning every bit as much as the words themselves.\n\nIf I break that structure into arbitrary pieces before indexing it, I shouldn't expect a retrieval engine—or an LLM—to reconstruct it perfectly later.\n\nA document isn't a sequence of tokens.\n\nIt's an argument made visible.\n\n*Next time I want to explore a different question: why finding a document isn't the same thing as discovering knowledge.*", "url": "https://wpnews.pro/news/documents-aren-t-bags-of-chunks", "canonical_source": "https://dev.to/valerykot/documents-arent-bags-of-chunks-3cha", "published_at": "2026-07-12 20:30:41+00:00", "updated_at": "2026-07-12 20:44:08.126699+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "natural-language-processing", "ai-research", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/documents-aren-t-bags-of-chunks", "markdown": "https://wpnews.pro/news/documents-aren-t-bags-of-chunks.md", "text": "https://wpnews.pro/news/documents-aren-t-bags-of-chunks.txt", "jsonld": "https://wpnews.pro/news/documents-aren-t-bags-of-chunks.jsonld"}}