chunk-lite: token-aware text chunking for RAG pipelines that doesn't cut mid-sentence

Developer Tejas released chunk-lite, an open-source JavaScript library for token-aware text chunking in RAG pipelines that respects sentence and paragraph boundaries to avoid cutting mid-thought. The library returns start and end offsets for traceability, supports custom tokenizers, and includes 26 tests. It is available on GitHub and npm.

Package 5 in my small open-source series, and the first one with a direct RAG/AI angle. Every RAG pipeline starts the same way: take a long document, split it into pieces small enough to embed, and hope the split points don't wreck the meaning inside each piece. A naive fixed-size split has no idea where a sentence or paragraph actually ends, so it happily cuts a chunk in half mid-thought - and a chunk that starts "...and that's why the deadline was moved" with no antecedent for "that" embeds as a blurry average of two unrelated ideas. It retrieves worse for both. chunk-lite makes the boundary-aware version the default. js import { chunkText } from 'chunk-lite'; const chunks = chunkText longDocument, { maxTokens: 300, overlapTokens: 50, } ; chunks 0 ; // { text: "...", startOffset: 0, endOffset: 1180, tokenCount: 298, index: 0 } A few design notes: startOffset / endOffset into the source text, so you can always trace a retrieved chunk back to exactly where it came from - for citations, highlighting, or re-chunking later without losing the mapping. tokenCounter tiktoken, a model's own endpoint, whatever if you need exact counts.Fully typed, zero required runtime dependencies, 26 tests covering overlap correctness, sentence/paragraph boundaries, oversized-sentence fallback, and custom tokenizer injection. GitHub: https://github.com/tejas821/chunk-lite https://github.com/tejas821/chunk-lite npm: npm i chunk-lite Full reasoning on chunk boundaries and the overlap trade-off is in the repo's CASE STUDY.md. Feedback welcome, especially if you've hit a boundary case this doesn't handle well.