{"slug": "37-1-mib-s-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-in", "title": "37.1 MiB/s with Zero Token Overflows: Guaranteed Token Budgets and Table-Aware Chunking in text-chipper", "summary": "A developer released text-chipper v1.1.0, a Rust-based text chunking library for RAG pipelines that enforces strict token budgets while preserving Markdown structure. Benchmarked against seven popular chunking approaches across 12 workloads, the engine reached 37.1 MiB/s single-core geometric mean throughput with zero token overflows, keeping table rows attached to repeated headers and maintaining exact source offsets so that doc[start:end] always equals chunk.text. The release adds native Hugging Face tokenizer support via from_pretrained and multi-tokenizer guards to avoid GIL overhead and tokenizer mismatch.", "body_md": "In RAG pipelines, text chunking usually forces an uncomfortable trade-off:\n\n`tiktoken` loop) are quick, but they slice right through Markdown code blocks, split words in half, and orphan table rows from their column headers.`v1.1.0` (Markdown table-header repetition and zero-GIL Hugging Face tokenizers), and how the Rust engine achieves this speed without breaking source offsets.\n---\n## 1. The Benchmark: Speed + Strict Token Compliance + Structure\nWe benchmarked `text-chipper` against seven popular chunking approaches across 12 workloads (synthetic prose, Markdown, code, PubMed, finance, Wikipedia, chat logs, and stress cases) with a strict `cl100k_base`, `overlap=0`).\n### Throughput (Single-Core Geometric Mean across 12 Workloads)\n`TokenChunker`).` markdown_doc` and 3,434 in `code_heavy`).\nConsider a 200-row financial or inventory table in Markdown:\n\n```\n| Row | Item | Category | Price |\n| --- | --- | --- | ---: |\n| 001 | Product-1 | Hardware | $10 |\n...\n| 200 | Product-200 | Hardware | $2000 |\n```\n\nWhen a 512-token budget forces this table to split into multiple chunks, two bad things typically happen in standard splitters:\n\n`| 045 | Prod` in chunk 1, `uct-45 | Hardware |` in chunk 2).`| Row | Item | Category | Price |`), leaving the embedding model and LLM with floating rows of numbers that lack column names.\n### How `doc[start:end] == chunk.text`\nA core invariant of `chunk.text` is always an exact contiguous slice of the normalized input\n\n```\nassert doc[chunk.start_index : chunk.end_index] == chunk.text\n```\n\nIf we mutated `chunk.text` on chunk 2 by injecting the table header from byte 0 into a row slice at byte 4,000, source byte/character offsets for citations and UI highlighting would break.\n\nInstead, `text-chipper` treats the repeated table header as **structural context**, just like section headings (`chunk.headers`):\n\n`max_length`, it is never split.` R_ITEM`), so rows are kept whole whenever they fit.`| --- |` delimiter row (`chunk.table_header` is `None`).` chunk.contextualized_text`.\n\n``` python\nfrom text_chipper import TextChipper\nheader = \"| Row | Item | Category | Price |\\n| --- | --- | --- | ---: |\"\nrows = [f\"| {i:03d} | Product-{i} | Hardware | ${i * 10} |\" for i in range(1, 201)]\ndoc = \"# Inventory\\n\\n\" + header + \"\\n\" + \"\\n\".join(rows) + \"\\n\"\nchipper = TextChipper(max_length=120, overlap=0, encoding=\"cl100k_base\")\nchunks = chipper.chunk(doc)\nfor idx, c in enumerate(chunks):\n    assert c.token_count <= 120\n    assert chipper.count_tokens(c.contextualized_text) == c.token_count\n    assert header in c.contextualized_text          # Every chunk has the table header!\n    assert doc[c.start_index : c.end_index] == c.text  # And exact source offsets still hold!\n```\n\nIf you run local embedding models (`BAAI/bge-large-en-v1.5`, `nomic-embed-text`, `jina-embeddings`) or local LLMs (` Qwen`, `Llama-3`, `Mistral`), counting tokens with OpenAI's `cl100k_base` is dangerous: a 500-token chunk under `cl100k_base` can easily be 610+ tokens under your model's actual tokenizer.\n\nEven worse, if you pass a Python `AutoTokenizer` callback into a chunker, calling back into Python thousands of times per document acquires the Python GIL on every probe and destroys multi-threaded performance.\n\n`AutoTokenizer` and `from_pretrained`\nIn `text-chipper` v1.1.0:\n\n`TextChipper.from_pretrained(\"BAAI/bge-large-en-v1.5\", max_length=510)` directly (resolving `tokenizer.json` from a local folder/file or via `huggingface_hub` / `tokenizers` / `transformers`).` transformers.PreTrainedTokenizerFast` (`AutoTokenizer.from_pretrained(...)`) or `tokenizers.Tokenizer` via `tokenizer=`.`.backend_tokenizer.to_str()` and loads it directly into Hugging Face's native Rust `Backend::from_tokenizer_bytes`). `callable(text) -> int` or object with `.encode(text)`.\n### Multi-Tokenizer Guards (`guard_tokenizers`)\nWhat if your pipeline sends the same chunk to `bge-large-en-v1.5`) `bge-reranker-v2-m3`) or LLM with a different vocabulary?\n\n``` python\nfrom transformers import AutoTokenizer\nfrom text_chipper import TextChipper\nembed_tok = AutoTokenizer.from_pretrained(\"BAAI/bge-large-en-v1.5\")\nrerank_tok = AutoTokenizer.from_pretrained(\"BAAI/bge-reranker-v2-m3\")\nchipper = TextChipper(\n    max_length=510,  # leave 2 tokens for [CLS] and [SEP] under a 512 limit\n    overlap=32,\n    tokenizer=embed_tok,\n    guard_tokenizers=[rerank_tok],\n)\nfor chunk in chipper.chunk(markdown_text):\n    # Guaranteed <= 510 tokens under BOTH BGE and BGE-reranker-v2-m3!\n    print(chunk.token_count, chunk.contextualized_text)\n```\n\nWhen processing massive datasets or log exports, loading a 514.3 MiB file into a single Python string and calling `chunk()` spikes process memory by **4.6 GB**.\n\nBecause `text-chipper`'s `BlockParser` and `SmartChunker` are incremental line-driven state machines, `from_file_stream(path)` streams the 514.3 MiB file in **25 seconds** (20.5 MiB/s on a single core) with only **56 MB** of memory growth—while pruning its Unicode normalization offset map behind the active chunk watermark.\n\n`text-chipper` v1.1.0 is open source (Apache-2.0) and available now on PyPI and crates.io:\n\n```\npip install -U text-chipper\n```\n\n`text_chipper`):", "url": "https://wpnews.pro/news/37-1-mib-s-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-in", "canonical_source": "https://dev.to/kameh0use/371-mibs-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-chunking-in-34bl", "published_at": "2026-11-21 18:09:38+00:00", "updated_at": "2026-11-21 18:12:06.271481+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools"], "entities": ["text-chipper", "Hugging Face", "tiktoken", "BAAI/bge-large-en-v1.5", "cl100k_base"], "also_reported_by": [], "takeaway": "text-chipper v1.1.0 enforces exact token budgets per chunk while repeating Markdown table headers as structural context, preserving doc[start:end] == chunk.text offsets at 37.1 MiB/s single-core throughput.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/37-1-mib-s-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-in", "markdown": "https://wpnews.pro/news/37-1-mib-s-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-in.md", "text": "https://wpnews.pro/news/37-1-mib-s-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-in.txt", "jsonld": "https://wpnews.pro/news/37-1-mib-s-with-zero-token-overflows-guaranteed-token-budgets-and-table-aware-in.jsonld"}}