For now, I ran a few experiments and looked into this:
The distinction I found most useful is not simply which JSON file takes precedence, but what is serialized, which implementation is constructed, and what the caller asks that implementation to do. These are related but separate layers. That explains much of the overlap, and also why a saved setting can differ from the behavior of a particular AutoTokenizer call.
1. What is each tokenizer file for? tokenizer.json primarily serializes the low-level tokenizers pipeline: the subword model and vocabulary, normalizer, pre-tokenizer, post-processor, decoder, added tokens, and possibly backend padding/truncation settings. tokenizer_config.json primarily describes the Transformers-facing tokenizer: class or auto- hints, model-oriented limits, padding/truncation sides, special-token roles, and other wrapper-level settings. Other files may be necessary, including chat templates, older vocabulary formats, config.json, processor files, and model-specific sidecars.
2. What does tokenizer_config.json add beyond tokenizer.json? It can supply information the tokenization engine alone cannot infer: which Python class should load it, what the model considers BOS/EOS/PAD/UNK, which side to pad or truncate on, what maximum length is declared, and sometimes legacy chat-template metadata. A token being present in the backend vocabulary is not the same as its being assigned a model-facing role. The exact interpretation of individual keys depends on the class and Transformers release.
3. Why the overlap? Backend state, wrapper defaults, and per-call policy are different things: model_max_length does not activate truncation, backend padding does not automatically mean padding=True, and added-token records are not named BOS/EOS/PAD roles. Legacy compatibility explains some duplication. There is no universal whole-file override rule across classes and versions.
4. How is the tokenizer loaded? AutoTokenizer resolves files and selects a class using metadata, config.json, registrations and release-specific dispatch rules. That class may load or reconstruct stages and apply defaults, so inspect the effective runtime pipeline. For precise behavior see v5.22.0 dispatch and backend construction.
5. What happens at execution time, especially truncation? Low-level Tokenizer.from_file(...).encode(text) can follow truncation saved in tokenizer.json, whereas ordinary Transformers tok(text) follows the wrapper’s call-time policy and may not truncate. Use and verify tok(text, truncation=True, max_length=N) when a limit is required. A stored limit or model_max_length alone is not an enforcement guarantee.
For ordinary use, prefer the checkpoint’s documented AutoTokenizer or AutoProcessor route; pin compatible library and checkpoint versions; make truncation/padding options explicit; inspect the selected class, IDs and masks, and for chat the rendered prompt. Compare against raw JSON only as a diagnostic reference, not as an automatic reason to bypass model-specific logic.
A first, tokenizer-only inspection needs no model weights:
import transformers
from transformers import AutoTokenizer
repo = "your-org/your-model"
revision = "FULL_COMMIT_SHA" # replace with an actual pinned commit
tok = AutoTokenizer.from_pretrained(repo, revision=revision)
print("Transformers:", transformers.__version__)
print("class:", type(tok).__name__)
print("max length:", tok.model_max_length)
print("sides:", tok.padding_side, tok.truncation_side)
print("special roles:", tok.special_tokens_map)
text = "hello world\nBonjour 世界"
for label, kwargs in [("default", {}),
("explicit limit", {"truncation": True,
"max_length": 8})]:
ids = tok(text, **kwargs)["input_ids"]
print(label, len(ids), ids)
print("decoded:", repr(tok.decode(ids, skip_special_tokens=False)))
Replace the placeholders; the eight-token limit is illustrative and includes special tokens. Avoid automatically trusting unknown remote code.
The optional sections give file mappings, a truncation experiment and limited compatibility observations. The independent technical report documents the experiments; they do not imply that the original poster encountered those bugs.