{"slug": "chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesn-t-cut-mid", "title": "chunk-lite: token-aware text chunking for RAG pipelines that doesn't cut mid-sentence", "summary": "Developer Tejas released chunk-lite, an open-source JavaScript library for token-aware text chunking in RAG pipelines that respects sentence and paragraph boundaries to avoid cutting mid-thought. The library returns start and end offsets for traceability, supports custom tokenizers, and includes 26 tests. It is available on GitHub and npm.", "body_md": "Package #5 in my small open-source series, and the first one with a direct RAG/AI angle.\n\nEvery RAG pipeline starts the same way: take a long document, split it into pieces small enough to embed, and hope the split points don't wreck the meaning inside each piece. A naive fixed-size split has no idea where a sentence or paragraph actually ends, so it happily cuts a chunk in half mid-thought - and a chunk that starts \"...and that's why the deadline was moved\" with no antecedent for \"that\" embeds as a blurry average of two unrelated ideas. It retrieves worse for both.\n\n**chunk-lite** makes the boundary-aware version the default.\n\n``` js\nimport { chunkText } from 'chunk-lite';\n\nconst chunks = chunkText(longDocument, {\n  maxTokens: 300,\n  overlapTokens: 50,\n});\n\nchunks[0];\n// { text: \"...\", startOffset: 0, endOffset: 1180, tokenCount: 298, index: 0 }\n```\n\nA few design notes:\n\n`startOffset`\n\n/`endOffset`\n\ninto the source text, so you can always trace a retrieved chunk back to exactly where it came from - for citations, highlighting, or re-chunking later without losing the mapping.`tokenCounter`\n\n(tiktoken, a model's own endpoint, whatever) if you need exact counts.Fully typed, zero required runtime dependencies, 26 tests covering overlap correctness, sentence/paragraph boundaries, oversized-sentence fallback, and custom tokenizer injection.\n\nGitHub: [https://github.com/tejas821/chunk-lite](https://github.com/tejas821/chunk-lite)\n\nnpm: `npm i chunk-lite`\n\nFull reasoning on chunk boundaries and the overlap trade-off is in the repo's CASE_STUDY.md. Feedback welcome, especially if you've hit a boundary case this doesn't handle well.", "url": "https://wpnews.pro/news/chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesn-t-cut-mid", "canonical_source": "https://dev.to/tejas821/chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesnt-cut-mid-sentence-1424", "published_at": "2026-08-27 04:23:09+00:00", "updated_at": "2026-08-27 04:47:56.642739+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "machine-learning", "natural-language-processing"], "entities": ["chunk-lite", "Tejas", "GitHub", "npm"], "alternates": {"html": "https://wpnews.pro/news/chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesn-t-cut-mid", "markdown": "https://wpnews.pro/news/chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesn-t-cut-mid.md", "text": "https://wpnews.pro/news/chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesn-t-cut-mid.txt", "jsonld": "https://wpnews.pro/news/chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesn-t-cut-mid.jsonld"}}