In RAG pipelines, text chunking usually forces an uncomfortable trade-off:
tiktoken loop) are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers.v1.1.0 (Markdown table-header repetition and zero-GIL Hugging Face tokenizers), and how the Rust engine achieves this speed without breaking source offsets. #
1. The Benchmark: Speed + Strict Token Compliance + Structure #
We benchmarked text-chipper against seven popular chunking approaches across 12 workloads (synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases) with a strict cl100k_base, overlap=0).
Throughput (Single-Core Geometric Mean across 12 Workloads)
TokenChunker). markdown_doc and 3,434 in code_heavy).
Consider a 200-row financial or inventory table in Markdown:
| Row | Item | Category | Price |
| --- | --- | --- | ---: |
| 001 | Product-1 | Hardware | $10 |
...
| 200 | Product-200 | Hardware | $2000 |
When a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters:
| 045 | Prod in chunk 1, uct-45 | Hardware | in chunk 2).| Row | Item | Category | Price |), leaving the embedding model and LLM with floating rows of numbers that lack column names.
How doc[start:end] == chunk.text
A core invariant of chunk.text is always an exact contiguous slice of the normalized input
assert doc[chunk.start_index : chunk.end_index] == chunk.text
If we mutated chunk.text on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break.
Instead, text-chipper treats the repeated table header as structural context, just like section headings (chunk.headers):
max_length, it is never split. R_ITEM), so rows are kept whole whenever they fit.| --- | delimiter row (chunk.table_header is None). chunk.contextualized_text.
from text_chipper import TextChipper
header = "| Row | Item | Category | Price |\n| --- | --- | --- | ---: |"
rows = [f"| {i:03d} | Product-{i} | Hardware | ${i * 10} |" for i in range(1, 201)]
doc = "# Inventory\n\n" + header + "\n" + "\n".join(rows) + "\n"
chipper = TextChipper(max_length=120, overlap=0, encoding="cl100k_base")
chunks = chipper.chunk(doc)
for idx, c in enumerate(chunks):
assert c.token_count <= 120
assert chipper.count_tokens(c.contextualized_text) == c.token_count
assert header in c.contextualized_text # Every chunk has the table header!
assert doc[c.start_index : c.end_index] == c.text # And exact source offsets still hold!
If you run local embedding models (BAAI/bge-large-en-v1.5, nomic-embed-text, jina-embeddings) or local LLMs ( Qwen, Llama-3, Mistral), counting tokens with OpenAI's cl100k_base is dangerous: a 500-token chunk under cl100k_base can easily be 610+ tokens under your model's actual tokenizer.
Even worse, if you pass a Python AutoTokenizer callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance.
AutoTokenizer and from_pretrained
In text-chipper v1.1.0:
TextChipper.from_pretrained("BAAI/bge-large-en-v1.5", max_length=510) directly (resolving tokenizer.json from a local folder/file or via huggingface_hub / tokenizers / transformers). transformers.PreTrainedTokenizerFast (AutoTokenizer.from_pretrained(...)) or tokenizers.Tokenizer via tokenizer=..backend_tokenizer.to_str() and loads it directly into Hugging Face's native Rust Backend::from_tokenizer_bytes). callable(text) -> int or object with .encode(text).
Multi-Tokenizer Guards (guard_tokenizers)
What if your pipeline sends the same chunk to bge-large-en-v1.5) bge-reranker-v2-m3) or LLM with a different vocabulary?
from transformers import AutoTokenizer
from text_chipper import TextChipper
embed_tok = AutoTokenizer.from_pretrained("BAAI/bge-large-en-v1.5")
rerank_tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3")
chipper = TextChipper(
max_length=510, # leave 2 tokens for [CLS] and [SEP] under a 512 limit
overlap=32,
tokenizer=embed_tok,
guard_tokenizers=[rerank_tok],
)
for chunk in chipper.chunk(markdown_text):
print(chunk.token_count, chunk.contextualized_text)
When processing massive datasets or log exports, a 514.3 MiB file into a single Python string and calling chunk() spikes process memory by 4.6 GB.
Because text-chipper's BlockParser and SmartChunker are incremental line-driven state machines, from_file_stream(path) streams the 514.3 MiB file in 25 seconds (20.5 MiB/s on a single core) with only 56 MB of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark.
text-chipper v1.1.0 is open source (Apache-2.0) and available now on PyPI and crates.io:
pip install -U text-chipper
text_chipper):