# 37.1 MiB/s with Zero Token Overflows: Guaranteed Token Budgets and Table-Aware Chunking in text-chipper

> Source: <https://dev.to/kameh0use/371-mibs-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-chunking-in-34bl>
> Published: 2026-11-21 18:09:38+00:00

**Key takeaway:** text-chipper v1.1.0 enforces exact token budgets per chunk while repeating Markdown table headers as structural context, preserving doc[start:end] == chunk.text offsets at 37.1 MiB/s single-core throughput.

In RAG pipelines, text chunking usually forces an uncomfortable trade-off:

`tiktoken` loop) are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers.`v1.1.0` (Markdown table-header repetition and zero-GIL Hugging Face tokenizers), and how the Rust engine achieves this speed without breaking source offsets.
---
## 1. The Benchmark: Speed + Strict Token Compliance + Structure
We benchmarked `text-chipper` against seven popular chunking approaches across 12 workloads (synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases) with a strict `cl100k_base`, `overlap=0`).
### Throughput (Single-Core Geometric Mean across 12 Workloads)
`TokenChunker`).` markdown_doc` and 3,434 in `code_heavy`).
Consider a 200-row financial or inventory table in Markdown:

```
| Row | Item | Category | Price |
| --- | --- | --- | ---: |
| 001 | Product-1 | Hardware | $10 |
...
| 200 | Product-200 | Hardware | $2000 |
```

When a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters:

`| 045 | Prod` in chunk 1, `uct-45 | Hardware |` in chunk 2).`| Row | Item | Category | Price |`), leaving the embedding model and LLM with floating rows of numbers that lack column names.
### How `doc[start:end] == chunk.text`
A core invariant of `chunk.text` is always an exact contiguous slice of the normalized input

```
assert doc[chunk.start_index : chunk.end_index] == chunk.text
```

If we mutated `chunk.text` on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break.

Instead, `text-chipper` treats the repeated table header as **structural context**, just like section headings (`chunk.headers`):

`max_length`, it is never split.` R_ITEM`), so rows are kept whole whenever they fit.`| --- |` delimiter row (`chunk.table_header` is `None`).` chunk.contextualized_text`.

``` python
from text_chipper import TextChipper
header = "| Row | Item | Category | Price |\n| --- | --- | --- | ---: |"
rows = [f"| {i:03d} | Product-{i} | Hardware | ${i * 10} |" for i in range(1, 201)]
doc = "# Inventory\n\n" + header + "\n" + "\n".join(rows) + "\n"
chipper = TextChipper(max_length=120, overlap=0, encoding="cl100k_base")
chunks = chipper.chunk(doc)
for idx, c in enumerate(chunks):
    assert c.token_count <= 120
    assert chipper.count_tokens(c.contextualized_text) == c.token_count
    assert header in c.contextualized_text          # Every chunk has the table header!
    assert doc[c.start_index : c.end_index] == c.text  # And exact source offsets still hold!
```

If you run local embedding models (`BAAI/bge-large-en-v1.5`, `nomic-embed-text`, `jina-embeddings`) or local LLMs (` Qwen`, `Llama-3`, `Mistral`), counting tokens with OpenAI's `cl100k_base` is dangerous: a 500-token chunk under `cl100k_base` can easily be 610+ tokens under your model's actual tokenizer.

Even worse, if you pass a Python `AutoTokenizer` callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance.

`AutoTokenizer` and `from_pretrained`
In `text-chipper` v1.1.0:

`TextChipper.from_pretrained("BAAI/bge-large-en-v1.5", max_length=510)` directly (resolving `tokenizer.json` from a local folder/file or via `huggingface_hub` / `tokenizers` / `transformers`).` transformers.PreTrainedTokenizerFast` (`AutoTokenizer.from_pretrained(...)`) or `tokenizers.Tokenizer` via `tokenizer=`.`.backend_tokenizer.to_str()` and loads it directly into Hugging Face's native Rust `Backend::from_tokenizer_bytes`). `callable(text) -> int` or object with `.encode(text)`.
### Multi-Tokenizer Guards (`guard_tokenizers`)
What if your pipeline sends the same chunk to `bge-large-en-v1.5`) `bge-reranker-v2-m3`) or LLM with a different vocabulary?

``` python
from transformers import AutoTokenizer
from text_chipper import TextChipper
embed_tok = AutoTokenizer.from_pretrained("BAAI/bge-large-en-v1.5")
rerank_tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-v2-m3")
chipper = TextChipper(
    max_length=510,  # leave 2 tokens for [CLS] and [SEP] under a 512 limit
    overlap=32,
    tokenizer=embed_tok,
    guard_tokenizers=[rerank_tok],
)
for chunk in chipper.chunk(markdown_text):
    # Guaranteed <= 510 tokens under BOTH BGE and BGE-reranker-v2-m3!
    print(chunk.token_count, chunk.contextualized_text)
```

When processing massive datasets or log exports, loading a 514.3 MiB file into a single Python string and calling `chunk()` spikes process memory by **4.6 GB**.

Because `text-chipper`'s `BlockParser` and `SmartChunker` are incremental line-driven state machines, `from_file_stream(path)` streams the 514.3 MiB file in **25 seconds** (20.5 MiB/s on a single core) with only **56 MB** of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark.

`text-chipper` v1.1.0 is open source (Apache-2.0) and available now on PyPI and crates.io:

```
pip install -U text-chipper
```

`text_chipper`):
