# What is the purpose of each Hugging Face tokenizer file, and what does tokenizer_config.json add beyond tokenizer.json?

> Source: <https://discuss.huggingface.co/t/what-is-the-purpose-of-each-hugging-face-tokenizer-file-and-what-does-tokenizer-config-json-add-beyond-tokenizer-json/191091#post_2>
> Published: 2026-10-30 01:30:12+00:00

For now, I ran a few experiments and looked into this:

The distinction I found most useful is not simply **which JSON file takes precedence**, but **what is serialized, which implementation is constructed, and what the caller asks that implementation to do**. These are related but separate layers. That explains much of the overlap, and also why a saved setting can differ from the behavior of a particular `AutoTokenizer` call.

**1. What is each tokenizer file for?** `tokenizer.json` primarily serializes the low-level [`tokenizers` pipeline](https://huggingface.co/docs/tokenizers/api/tokenizer): the subword model and vocabulary, normalizer, pre-tokenizer, post-processor, decoder, added tokens, and possibly backend padding/truncation settings. `tokenizer_config.json` primarily describes the **Transformers-facing tokenizer**: class or auto-loading hints, model-oriented limits, padding/truncation sides, special-token roles, and other wrapper-level settings. Other files may be necessary, including chat templates, older vocabulary formats, `config.json`, processor files, and model-specific sidecars.

**2. What does `tokenizer_config.json` add beyond `tokenizer.json`?** It can supply information the tokenization engine alone cannot infer: which Python class should load it, what the model considers BOS/EOS/PAD/UNK, which side to pad or truncate on, what maximum length is declared, and sometimes legacy chat-template metadata. A token being present in the backend vocabulary is not the same as its being assigned a model-facing role. The exact interpretation of individual keys depends on the class and Transformers release.

**3. Why the overlap?** Backend state, wrapper defaults, and per-call policy are different things: `model_max_length` does not activate truncation, backend padding does not automatically mean `padding=True`, and added-token records are not named BOS/EOS/PAD roles. Legacy compatibility explains some duplication. There is no universal whole-file override rule across classes and versions.

**4. How is the tokenizer loaded?** `AutoTokenizer` resolves files and selects a class using metadata, `config.json`, registrations and release-specific dispatch rules. That class may load or reconstruct stages and apply defaults, so inspect the **effective runtime pipeline**. For precise behavior see [v5.22.0 dispatch](https://github.com/huggingface/transformers/blob/v5.22.0/src/transformers/models/auto/tokenization_auto.py) and [backend construction](https://github.com/huggingface/transformers/blob/v5.22.0/src/transformers/tokenization_utils_tokenizers.py).

**5. What happens at execution time, especially truncation?** Low-level `Tokenizer.from_file(...).encode(text)` can follow truncation saved in `tokenizer.json`, whereas ordinary Transformers `tok(text)` follows the wrapper’s call-time policy and may not truncate. Use and verify `tok(text, truncation=True, max_length=N)` when a limit is required. A stored limit or `model_max_length` alone is not an enforcement guarantee.

For ordinary use, prefer the checkpoint’s **documented `AutoTokenizer` or `AutoProcessor` route**; pin compatible library and checkpoint versions; make truncation/padding options explicit; inspect the selected class, IDs and masks, and for chat the rendered prompt. Compare against raw JSON only as a **diagnostic reference**, not as an automatic reason to bypass model-specific logic.

A first, tokenizer-only inspection needs no model weights:

``` python
import transformers
from transformers import AutoTokenizer

repo = "your-org/your-model"
revision = "FULL_COMMIT_SHA"  # replace with an actual pinned commit

tok = AutoTokenizer.from_pretrained(repo, revision=revision)
print("Transformers:", transformers.__version__)
print("class:", type(tok).__name__)
print("max length:", tok.model_max_length)
print("sides:", tok.padding_side, tok.truncation_side)
print("special roles:", tok.special_tokens_map)

text = "hello  world\nBonjour 世界"
for label, kwargs in [("default", {}),
                      ("explicit limit", {"truncation": True,
                                          "max_length": 8})]:
    ids = tok(text, **kwargs)["input_ids"]
    print(label, len(ids), ids)
    print("decoded:", repr(tok.decode(ids, skip_special_tokens=False)))
```

Replace the placeholders; the eight-token limit is illustrative and includes special tokens. Avoid automatically trusting unknown remote code.

The optional sections give file mappings, a truncation experiment and limited compatibility observations. The [independent technical report](https://huggingface.co/datasets/John6666/forum3/blob/main/transformers_v5_drift/transformers_v5_drift_technical_report_20261030.md) documents the experiments; they do **not** imply that the original poster encountered those bugs.