tok.apply_chat_template(msgs, add_generation_prompt=True)
I have copied that line more times than I can count without thinking about what any of it does. The official example writes it that way, the output looks right, you move on.
Then you have to build your own SFT data and compute your own loss mask, and it turns out every piece of that line is load-bearing. I spent today pulling it apart. Notes below.
The counterintuitive part first: the model has no idea messages
exists.
messages
is for Python. Roles are real tokens in the vocabulary.tokenizer_config.json
or as a standalone chat_template.jinja
. When both exist the file wins.add_generation_prompt=False
. Inference renders msgs[:-1]
with add_generation_prompt=True
.messages
exists I had some vague picture in my head where the role was structured metadata and something inside the model read it. Nope.
[{"role": "user", "content": "What is the Young's modulus of graphene?"}]
That list lives in your Python process and nowhere else. What reaches the model is:
[gMASK]<sop><|user|>
What is the Young's modulus of graphene?<|assistant|>
and after tokenization:
[151331, 151333, 151336, 198, ...content ids..., 151337, 198]
Those leading numbers are GLM-4's ids for [gMASK]
, <sop>
, <|user|>
and <|assistant|>
. Every model family has its own, so don't hardcode them across models.
The role is a token, not a field. The model knows it is its turn to talk because it sees id 151337
, the same way it learns any other pattern. No JSON at inference time, no key lookup, no schema. One sequence.
Nearly everything below follows from that.
Jinja2 is a text templating engine originally built to render HTML for Flask. Three constructs, that's it:
| Syntax | Purpose |
|---|---|
{{ expr }} |
|
| print the value of an expression | |
{% ... %} |
|
control flow: for , if , set |
|
{# ... #} |
|
| comment, produces no output |
HuggingFace borrowed it for one job: flatten structured messages into the string the model was trained on, and flatten it the same way every time. The minimal shape:
{%- for message in messages %}
{{- '<|' + message['role'] + '|>\n' + message['content'] + '<|endoftext|>\n' }}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|assistant|>\n' }}
{%- endif %}
{%-
and -%}
strip whitespace around the tag. Real templates are covered in them.
I assumed this was a formatting preference. It isn't. Templates get indented so humans can read them, and without stripping, every newline and indent in the source lands verbatim in the prompt. One extra \n
and the token sequence no longer matches what training saw.
If you write your own, print the result with repr()
rather than trusting your eyes.
transformers
renders in a jinja2.sandbox.ImmutableSandboxedEnvironment
, so arbitrary attribute access and side effects are blocked. You can't call random Python from in there. What you get:
messages
, plus tools
when tool definitions are passedadd_generation_prompt
bos_token
, eos_token
and friends, from the tokenizerraise_exception()
, for templates that reject malformed conversations, like a system message in the wrong slotstrftime_now()
, for templates that inject the current dateTwo places, same string.
The old way is inline in tokenizer_config.json
. That's the file AutoTokenizer.from_pretrained()
reads, and note it does not hold the vocabulary. That's in tokenizer.json
, or vocab.json
plus merges.txt
, or a SentencePiece *.model
. It only records how to construct the tokenizer object:
{
"tokenizer_class": "PreTrainedTokenizerFast",
"model_max_length": 131072,
"padding_side": "left",
"bos_token": "[gMASK]",
"eos_token": "<|endoftext|>",
"pad_token": "<|endoftext|>",
"clean_up_tokenization_spaces": false,
"added_tokens_decoder": {
"151329": { "content": "<|endoftext|>", "special": true, "normalized": false },
"151336": { "content": "<|user|>", "special": true, "normalized": false }
},
"chat_template": "{%- for message in messages %}..." // ← inline, one giant line
}
For SFT work, a few fields are worth a look.
added_tokens_decoder
decides whether <|user|>
is one atomic token or gets shredded into BPE pieces. Shredded, the role marker stops being a clean signal and the model has to infer the boundary from "<
then |
then user
". It can learn that. No reason to spend the capacity.
padding_side
is right for training and must be left for batched generation. In a right-padded generation batch every sequence shorter than the longest one comes out garbage.
eos_token
and pad_token
govern both where generation stops and which positions get carved out of the loss.
The new way is a standalone chat_template.jinja
, which newer transformers
versions write by default from save_pretrained()
. The reason is mundane: inside JSON every newline is \n
and every quote is escaped, so a 200-line template becomes one unreadable line with no diff and no highlighting.
The file wins on precedence. Plenty of repos ship both for backward compatibility, which leaves a trap. Once the two drift apart, behavior depends on your transformers
version, and that is a miserable thing to track down. Check they agree before you debug anything else.
This is where I actually got stuck today. One SFT sample:
msgs = [
{"role": "user", "content": "What is the Young's modulus of graphene?"},
{"role": "assistant", "content": "About 1 TPa."},
]
At training time you have the answer. That's the supervision signal, so you render the whole thing with no generation prompt:
tok.apply_chat_template(msgs, add_generation_prompt=False)
<|user|>
What is the Young's modulus of graphene?<|assistant|>
About 1 TPa.<|endoftext|>
Setting it to True
would append a second, empty <|assistant|>\n
after the answer. A dangling role marker, which the model would learn to emit.
At inference you don't have the answer. Producing it is the point, so you pass msgs[:-1]
:
tok.apply_chat_template(msgs[:-1], add_generation_prompt=True)
<|user|>
What is the Young's modulus of graphene?<|assistant|>
add_generation_prompt=True
is what appends that trailing <|assistant|>\n
. Without it the last token the model sees is ?
, and it will cheerfully continue the user's turn, inventing a follow-up question instead of answering. With it, the model is standing exactly where, during training, the next token was the start of an answer.
Line the two up:
training: <|user|>\nWhat is ... graphene?<|assistant|>\n | About 1 TPa.<|endoftext|>
inference: <|user|>\nWhat is ... graphene?<|assistant|>\n | ← generation starts here
└──────────── must match exactly ───────────┘
Everything left of the bar is the model's conditioning context. Training taught it "given this prefix, emit About
". If inference rebuilds that prefix with one extra newline, or a space the training path didn't have, the model is conditioning on something outside its training distribution.
What makes it nasty is that it degrades invisibly. Print both strings and they look identical. Nothing in the logs.
So just assert:
full = tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=False)
prefix = tok.apply_chat_template(msgs[:-1], tokenize=True, add_generation_prompt=True)
assert full[:len(prefix)] == prefix
Three lines, and it hands you the loss-mask boundary for free: len(prefix)
is where the answer starts.
Roles are tokens, not fields, so there are no field boundaries to slice on. You locate the assistant span yourself.
The direct approach is the length difference:
labels = [-100] * len(prefix) + full[len(prefix):]
The boundary lands naturally after <|assistant|>\n
. Clean.
The cleverer-looking alternative is to regex the rendered text for <|assistant|>
and map character offsets back to token indices. Don't. The moment a message's content
legitimately contains that string it's wrong, and wrong silently.
The length-diff trick has one real limit: it only handles the last turn. For multi-turn data where you want loss on every assistant reply, you'd render incrementally turn by turn, which gets ugly fast.
The proper fix is to mark the assistant span in the template itself:
{%- if message['role'] == 'assistant' %}
{% generation %}{{- message['content'] + '<|endoftext|>' }}{% endgeneration %}
{%- endif %}
Then ask for the mask:
out = tok.apply_chat_template(
msgs, tokenize=True, return_dict=True,
return_assistant_tokens_mask=True,
)
mask = out["assistant_masks"] # 1 on assistant tokens
Correct for any number of turns. The cost is that the template has to contain {% generation %}
blocks, and a lot of published templates don't, which is why I haven't switched to it yet. Check first, or add the blocks to your own copy.
Double BOS. The template already emitted BOS. Call tok(text)
after it and the default add_special_tokens=True
adds another:
text = tok.apply_chat_template(msgs, tokenize=False)
ids = tok(text, add_special_tokens=False)["input_ids"] # ← required
Or skip the round trip with tokenize=True
, which uses add_special_tokens=False
internally anyway.
Hand-built strings. Writing f"<|user|>\n{q}<|assistant|>\n"
in your data pipeline works, right up until the official template changes, or you swap base models, or someone adds a system prompt. Let the template be the single source of truth and stop thinking about it.
Inconsistent template branches. Some hand-edited templates emit <|assistant|>\n
inside the loop and <|assistant|>
in the add_generation_prompt
branch, one newline short. The official GLM and Qwen templates are fine, but if you've touched a template, or added tool definitions or a system prompt, that assert is the only thing that will tell you.
Special tokens in user content. Jinja does plain string concatenation with no escaping. A content
field containing the literal text <|assistant|>
tokenizes into the real special token, and the model reads a genuine turn boundary. Prompt injection, training-data edition. If your corpus is scraped or model-generated, scan it first.
It comes down to one thing: the byte string you train on and the byte string you infer on have to share an identical prefix. Jinja2 produces that string, tokenizer_config.json
and chat_template.jinja
hold the recipe, and add_generation_prompt
is the only difference the two paths should have.
Honestly, none of this comes up most of the time. Use a stock model on a standard pipeline and apply_chat_template
handles everything; you never need to know what's underneath. It only surfaces when you start building your own data, computing your own masks, or swapping base models. And when it does surface it doesn't throw. It just makes things slightly worse, which is exactly why it's worth knowing in advance.
There's an isomorphic problem on the serving side. The QPS-as-load-signal mistake in Little's Law and vLLM autoscaling has the same shape: a metric that looks reasonable and is quietly measuring something else.
Further reading: the chat templating guide in the transformers
docs, and the whitespace-control section of the Jinja2 template designer reference.