cl100k

ByteLex reported that a tokenizer-free coordinate map built from 11 tokenizer vocabularies over 3-byte windows predicted its 237M-parameter byte-level language model's per-word errors with a Spearman …

A developer measured the token cost of injecting MCP tool definitions into a model's context, finding that the live discovery round trip is nearly free while the standing per-call cost of tool definit…

A new arXiv study (2608.21384v1) finds that Ukrainian and other Cyrillic-script languages face 68-121% token overhead on modern tokenizers and 220% on the older cl100k, compared to English, across nin…