cd /news/ai-tools/37-1-mib-s-with-zero-token-overflows… · home › topics › ai-tools › article
[ARTICLE · art-173857] src=dev.to ↗ pub= topic=ai-tools verified=true sentiment=↑ positive

37.1 MiB/s with Zero Token Overflows: Guaranteed Token Budgets and Table-Aware Chunking in text-chipper

A developer released text-chipper v1.1.0, a Rust-based text chunking library for RAG pipelines that enforces strict token budgets while preserving Markdown structure. Benchmarked against seven popular chunking approaches across 12 workloads, the engine reached 37.1 MiB/s single-core geometric mean throughput with zero token overflows, keeping table rows attached to repeated headers and maintaining exact source offsets so that doc[start:end] always equals chunk.text. The release adds native Hugging Face tokenizer support via from_pretrained and multi-tokenizer guards to avoid GIL overhead and tokenizer mismatch.

key takeaway

text-chipper v1.1.0 enforces exact token budgets per chunk while repeating Markdown table headers as structural context, preserving doc[start:end] == chunk.text offsets at 37.1 MiB/s single-core throughput.

by read3 min views2 publishedNov 21, 2026

In RAG pipelines, text chunking usually forces an uncomfortable trade-off:

tiktoken loop) are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers.v1.1.0 (Markdown table-header repetition and zero-GIL Hugging Face tokenizers), and how the Rust engine achieves this speed without breaking source offsets. #

1. The Benchmark: Speed + Strict Token Compliance + Structure #

We benchmarked text-chipper against seven popular chunking approaches across 12 workloads (synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases) with a strict cl100k_base, overlap=0).

Throughput (Single-Core Geometric Mean across 12 Workloads)

TokenChunker). markdown_doc and 3,434 in code_heavy). Consider a 200-row financial or inventory table in Markdown:

| Row | Item | Category | Price |
| --- | --- | --- | ---: |
| 001 | Product-1 | Hardware | $10 |
...
| 200 | Product-200 | Hardware | $2000 |

When a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters:

| 045 | Prod in chunk 1, uct-45 | Hardware | in chunk 2).| Row | Item | Category | Price |), leaving the embedding model and LLM with floating rows of numbers that lack column names.

How doc[start:end] == chunk.text

A core invariant of chunk.text is always an exact contiguous slice of the normalized input

assert doc[chunk.start_index : chunk.end_index] == chunk.text

If we mutated chunk.text on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break.

Instead, text-chipper treats the repeated table header as structural context, just like section headings (chunk.headers):

max_length, it is never split. R_ITEM), so rows are kept whole whenever they fit.| --- | delimiter row (chunk.table_header is None). chunk.contextualized_text.

from text_chipper import TextChipper
header = "| Row | Item | Category | Price |\n| --- | --- | --- | ---: |"
rows = [f"| {i:03d} | Product-{i} | Hardware | ${i * 10} |" for i in range(1, 201)]
doc = "# Inventory\n\n" + header + "\n" + "\n".join(rows) + "\n"
chipper = TextChipper(max_length=120, overlap=0, encoding="cl100k_base")
chunks = chipper.chunk(doc)
for idx, c in enumerate(chunks):
    assert c.token_count <= 120
    assert chipper.count_tokens(c.contextualized_text) == c.token_count
    assert header in c.contextualized_text          # Every chunk has the table header!
    assert doc[c.start_index : c.end_index] == c.text  # And exact source offsets still hold!

If you run local embedding models (BAAI/bge-large-en-v1.5, nomic-embed-text, jina-embeddings) or local LLMs ( Qwen, Llama-3, Mistral), counting tokens with OpenAI's cl100k_base is dangerous: a 500-token chunk under cl100k_base can easily be 610+ tokens under your model's actual tokenizer.

Even worse, if you pass a Python AutoTokenizer callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance.

AutoTokenizer and from_pretrained In text-chipper v1.1.0:

TextChipper.from_pretrained("BAAI/bge-large-en-v1.5", max_length=510) directly (resolving tokenizer.json from a local folder/file or via huggingface_hub / tokenizers / transformers). transformers.PreTrainedTokenizerFast (AutoTokenizer.from_pretrained(...)) or tokenizers.Tokenizer via tokenizer=..backend_tokenizer.to_str() and loads it directly into Hugging Face's native Rust Backend::from_tokenizer_bytes). callable(text) -> int or object with .encode(text).

Multi-Tokenizer Guards (guard_tokenizers)

What if your pipeline sends the same chunk to bge-large-en-v1.5) bge-reranker-v2-m3) or LLM with a different vocabulary?

from transformers import AutoTokenizer
from text_chipper import TextChipper
embed_tok = AutoTokenizer.from_pretrained("BAAI/bge-large-en-v1.5")
rerank_tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3")
chipper = TextChipper(
    max_length=510,  # leave 2 tokens for [CLS] and [SEP] under a 512 limit
    overlap=32,
    tokenizer=embed_tok,
    guard_tokenizers=[rerank_tok],
)
for chunk in chipper.chunk(markdown_text):
    print(chunk.token_count, chunk.contextualized_text)

When processing massive datasets or log exports, a 514.3 MiB file into a single Python string and calling chunk() spikes process memory by 4.6 GB.

Because text-chipper's BlockParser and SmartChunker are incremental line-driven state machines, from_file_stream(path) streams the 514.3 MiB file in 25 seconds (20.5 MiB/s on a single core) with only 56 MB of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark.

text-chipper v1.1.0 is open source (Apache-2.0) and available now on PyPI and crates.io:

pip install -U text-chipper

text_chipper):

── more in #ai-tools 4 stories · sorted by recency
── more on @text-chipper 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/37-1-mib-s-with-zero…] indexed:0 read:3min 2026-11-21 · —