37.1 MiB/s with Zero Token Overflows: Guaranteed Token Budgets and Table-Aware Chunking in text-chipper A developer released text-chipper v1.1.0, a Rust-based text chunking library for RAG pipelines that enforces strict token budgets while preserving Markdown structure. Benchmarked against seven popular chunking approaches across 12 workloads, the engine reached 37.1 MiB/s single-core geometric mean throughput with zero token overflows, keeping table rows attached to repeated headers and maintaining exact source offsets so that doc[start:end] always equals chunk.text. The release adds native Hugging Face tokenizer support via from_pretrained and multi-tokenizer guards to avoid GIL overhead and tokenizer mismatch. In RAG pipelines, text chunking usually forces an uncomfortable trade-off: tiktoken loop are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers. v1.1.0 Markdown table-header repetition and zero-GIL Hugging Face tokenizers , and how the Rust engine achieves this speed without breaking source offsets. --- 1. The Benchmark: Speed + Strict Token Compliance + Structure We benchmarked text-chipper against seven popular chunking approaches across 12 workloads synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases with a strict cl100k base , overlap=0 . Throughput Single-Core Geometric Mean across 12 Workloads TokenChunker . markdown doc and 3,434 in code heavy . Consider a 200-row financial or inventory table in Markdown: | Row | Item | Category | Price | | --- | --- | --- | ---: | | 001 | Product-1 | Hardware | $10 | ... | 200 | Product-200 | Hardware | $2000 | When a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters: | 045 | Prod in chunk 1, uct-45 | Hardware | in chunk 2 . | Row | Item | Category | Price | , leaving the embedding model and LLM with floating rows of numbers that lack column names. How doc start:end == chunk.text A core invariant of chunk.text is always an exact contiguous slice of the normalized input assert doc chunk.start index : chunk.end index == chunk.text If we mutated chunk.text on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break. Instead, text-chipper treats the repeated table header as structural context , just like section headings chunk.headers : max length , it is never split. R ITEM , so rows are kept whole whenever they fit. | --- | delimiter row chunk.table header is None . chunk.contextualized text . python from text chipper import TextChipper header = "| Row | Item | Category | Price |\n| --- | --- | --- | ---: |" rows = f"| {i:03d} | Product-{i} | Hardware | ${i 10} |" for i in range 1, 201 doc = " Inventory\n\n" + header + "\n" + "\n".join rows + "\n" chipper = TextChipper max length=120, overlap=0, encoding="cl100k base" chunks = chipper.chunk doc for idx, c in enumerate chunks : assert c.token count <= 120 assert chipper.count tokens c.contextualized text == c.token count assert header in c.contextualized text Every chunk has the table header assert doc c.start index : c.end index == c.text And exact source offsets still hold If you run local embedding models BAAI/bge-large-en-v1.5 , nomic-embed-text , jina-embeddings or local LLMs Qwen , Llama-3 , Mistral , counting tokens with OpenAI's cl100k base is dangerous: a 500-token chunk under cl100k base can easily be 610+ tokens under your model's actual tokenizer. Even worse, if you pass a Python AutoTokenizer callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance. AutoTokenizer and from pretrained In text-chipper v1.1.0: TextChipper.from pretrained "BAAI/bge-large-en-v1.5", max length=510 directly resolving tokenizer.json from a local folder/file or via huggingface hub / tokenizers / transformers . transformers.PreTrainedTokenizerFast AutoTokenizer.from pretrained ... or tokenizers.Tokenizer via tokenizer= . .backend tokenizer.to str and loads it directly into Hugging Face's native Rust Backend::from tokenizer bytes . callable text - int or object with .encode text . Multi-Tokenizer Guards guard tokenizers What if your pipeline sends the same chunk to bge-large-en-v1.5 bge-reranker-v2-m3 or LLM with a different vocabulary? python from transformers import AutoTokenizer from text chipper import TextChipper embed tok = AutoTokenizer.from pretrained "BAAI/bge-large-en-v1.5" rerank tok = AutoTokenizer.from pretrained "BAAI/bge-reranker-v2-m3" chipper = TextChipper max length=510, leave 2 tokens for CLS and SEP under a 512 limit overlap=32, tokenizer=embed tok, guard tokenizers= rerank tok , for chunk in chipper.chunk markdown text : Guaranteed <= 510 tokens under BOTH BGE and BGE-reranker-v2-m3 print chunk.token count, chunk.contextualized text When processing massive datasets or log exports, loading a 514.3 MiB file into a single Python string and calling chunk spikes process memory by 4.6 GB . Because text-chipper 's BlockParser and SmartChunker are incremental line-driven state machines, from file stream path streams the 514.3 MiB file in 25 seconds 20.5 MiB/s on a single core with only 56 MB of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark. text-chipper v1.1.0 is open source Apache-2.0 and available now on PyPI and crates.io: pip install -U text-chipper text chipper :