# chunk-lite: token-aware text chunking for RAG pipelines that doesn't cut mid-sentence

> Source: <https://dev.to/tejas821/chunk-lite-token-aware-text-chunking-for-rag-pipelines-that-doesnt-cut-mid-sentence-1424>
> Published: 2026-08-27 04:23:09+00:00

Package #5 in my small open-source series, and the first one with a direct RAG/AI angle.

Every RAG pipeline starts the same way: take a long document, split it into pieces small enough to embed, and hope the split points don't wreck the meaning inside each piece. A naive fixed-size split has no idea where a sentence or paragraph actually ends, so it happily cuts a chunk in half mid-thought - and a chunk that starts "...and that's why the deadline was moved" with no antecedent for "that" embeds as a blurry average of two unrelated ideas. It retrieves worse for both.

**chunk-lite** makes the boundary-aware version the default.

``` js
import { chunkText } from 'chunk-lite';

const chunks = chunkText(longDocument, {
  maxTokens: 300,
  overlapTokens: 50,
});

chunks[0];
// { text: "...", startOffset: 0, endOffset: 1180, tokenCount: 298, index: 0 }
```

A few design notes:

`startOffset`

/`endOffset`

into the source text, so you can always trace a retrieved chunk back to exactly where it came from - for citations, highlighting, or re-chunking later without losing the mapping.`tokenCounter`

(tiktoken, a model's own endpoint, whatever) if you need exact counts.Fully typed, zero required runtime dependencies, 26 tests covering overlap correctness, sentence/paragraph boundaries, oversized-sentence fallback, and custom tokenizer injection.

GitHub: [https://github.com/tejas821/chunk-lite](https://github.com/tejas821/chunk-lite)

npm: `npm i chunk-lite`

Full reasoning on chunk boundaries and the overlap trade-off is in the repo's CASE_STUDY.md. Feedback welcome, especially if you've hit a boundary case this doesn't handle well.