Package #5 in my small open-source series, and the first one with a direct RAG/AI angle.
Every RAG pipeline starts the same way: take a long document, split it into pieces small enough to embed, and hope the split points don't wreck the meaning inside each piece. A naive fixed-size split has no idea where a sentence or paragraph actually ends, so it happily cuts a chunk in half mid-thought - and a chunk that starts "...and that's why the deadline was moved" with no antecedent for "that" embeds as a blurry average of two unrelated ideas. It retrieves worse for both.
chunk-lite makes the boundary-aware version the default.
import { chunkText } from 'chunk-lite';
const chunks = chunkText(longDocument, {
maxTokens: 300,
overlapTokens: 50,
});
chunks[0];
// { text: "...", startOffset: 0, endOffset: 1180, tokenCount: 298, index: 0 }
A few design notes:
startOffset
/endOffset
into the source text, so you can always trace a retrieved chunk back to exactly where it came from - for citations, highlighting, or re-chunking later without losing the mapping.tokenCounter
(tiktoken, a model's own endpoint, whatever) if you need exact counts.Fully typed, zero required runtime dependencies, 26 tests covering overlap correctness, sentence/paragraph boundaries, oversized-sentence fallback, and custom tokenizer injection.
GitHub: https://github.com/tejas821/chunk-lite
npm: npm i chunk-lite
Full reasoning on chunk boundaries and the overlap trade-off is in the repo's CASE_STUDY.md. Feedback welcome, especially if you've hit a boundary case this doesn't handle well.