# What Actually Gets Tokenized in SFT

> Source: <https://dev.to/jessiejia11/what-actually-gets-tokenized-in-sft-jbo>
> Published: 2026-08-13 07:17:00+00:00

`tok.apply_chat_template(msgs, add_generation_prompt=True)`

I have copied that line more times than I can count without thinking about what any of it does. The official example writes it that way, the output looks right, you move on.

Then you have to build your own SFT data and compute your own loss mask, and it turns out every piece of that line is load-bearing. I spent today pulling it apart. Notes below.

The counterintuitive part first: the model has no idea `messages`

exists.

`messages`

is for Python. Roles are real tokens in the vocabulary.`tokenizer_config.json`

or as a standalone `chat_template.jinja`

. When both exist the file wins.`add_generation_prompt=False`

. Inference renders `msgs[:-1]`

with `add_generation_prompt=True`

.`messages`

exists
I had some vague picture in my head where the role was structured metadata and something inside the model read it. Nope.

```
[{"role": "user", "content": "What is the Young's modulus of graphene?"}]
```

That list lives in your Python process and nowhere else. What reaches the model is:

```
[gMASK]<sop><|user|>
What is the Young's modulus of graphene?<|assistant|>
```

and after tokenization:

```
[151331, 151333, 151336, 198, ...content ids..., 151337, 198]
```

Those leading numbers are GLM-4's ids for `[gMASK]`

, `<sop>`

, `<|user|>`

and `<|assistant|>`

. Every model family has its own, so don't hardcode them across models.

The role is a token, not a field. The model knows it is its turn to talk because it sees id `151337`

, the same way it learns any other pattern. No JSON at inference time, no key lookup, no schema. One sequence.

Nearly everything below follows from that.

[Jinja2](https://jinja.palletsprojects.com/) is a text templating engine originally built to render HTML for Flask. Three constructs, that's it:

| Syntax | Purpose |
|---|---|
`{{ expr }}` |
print the value of an expression |
`{% ... %}` |
control flow: `for` , `if` , `set`
|
`{# ... #}` |
comment, produces no output |

HuggingFace borrowed it for one job: flatten structured messages into the string the model was trained on, and flatten it the same way every time. The minimal shape:

```
{%- for message in messages %}
    {{- '<|' + message['role'] + '|>\n' + message['content'] + '<|endoftext|>\n' }}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|assistant|>\n' }}
{%- endif %}
```

`{%-`

and `-%}`

strip whitespace around the tag. Real templates are covered in them.

I assumed this was a formatting preference. It isn't. Templates get indented so humans can read them, and without stripping, every newline and indent in the source lands verbatim in the prompt. One extra `\n`

and the token sequence no longer matches what training saw.

If you write your own, print the result with `repr()`

rather than trusting your eyes.

`transformers`

renders in a `jinja2.sandbox.ImmutableSandboxedEnvironment`

, so arbitrary attribute access and side effects are blocked. You can't call random Python from in there. What you get:

`messages`

, plus `tools`

when tool definitions are passed`add_generation_prompt`

`bos_token`

, `eos_token`

and friends, from the tokenizer`raise_exception()`

, for templates that reject malformed conversations, like a system message in the wrong slot`strftime_now()`

, for templates that inject the current dateTwo places, same string.

The old way is inline in `tokenizer_config.json`

. That's the file `AutoTokenizer.from_pretrained()`

reads, and note it does not hold the vocabulary. That's in `tokenizer.json`

, or `vocab.json`

plus `merges.txt`

, or a SentencePiece `*.model`

. It only records how to construct the tokenizer object:

```
{
  "tokenizer_class": "PreTrainedTokenizerFast",
  "model_max_length": 131072,
  "padding_side": "left",
  "bos_token": "[gMASK]",
  "eos_token": "<|endoftext|>",
  "pad_token": "<|endoftext|>",
  "clean_up_tokenization_spaces": false,
  "added_tokens_decoder": {
    "151329": { "content": "<|endoftext|>", "special": true, "normalized": false },
    "151336": { "content": "<|user|>",      "special": true, "normalized": false }
  },
  "chat_template": "{%- for message in messages %}..."   // ← inline, one giant line
}
```

For SFT work, a few fields are worth a look.

`added_tokens_decoder`

decides whether `<|user|>`

is one atomic token or gets shredded into BPE pieces. Shredded, the role marker stops being a clean signal and the model has to infer the boundary from "`<`

then `|`

then `user`

". It can learn that. No reason to spend the capacity.

`padding_side`

is right for training and must be left for batched generation. In a right-padded generation batch every sequence shorter than the longest one comes out garbage.

`eos_token`

and `pad_token`

govern both where generation stops and which positions get carved out of the loss.

The new way is a standalone `chat_template.jinja`

, which newer `transformers`

versions write by default from `save_pretrained()`

. The reason is mundane: inside JSON every newline is `\n`

and every quote is escaped, so a 200-line template becomes one unreadable line with no diff and no highlighting.

The file wins on precedence. Plenty of repos ship both for backward compatibility, which leaves a trap. Once the two drift apart, behavior depends on your `transformers`

version, and that is a miserable thing to track down. Check they agree before you debug anything else.

This is where I actually got stuck today. One SFT sample:

```
msgs = [
    {"role": "user",      "content": "What is the Young's modulus of graphene?"},
    {"role": "assistant", "content": "About 1 TPa."},
]
```

**At training time you have the answer.** That's the supervision signal, so you render the whole thing with no generation prompt:

```
tok.apply_chat_template(msgs, add_generation_prompt=False)
<|user|>
What is the Young's modulus of graphene?<|assistant|>
About 1 TPa.<|endoftext|>
```

Setting it to `True`

would append a second, empty `<|assistant|>\n`

after the answer. A dangling role marker, which the model would learn to emit.

**At inference you don't have the answer.** Producing it is the point, so you pass `msgs[:-1]`

:

```
tok.apply_chat_template(msgs[:-1], add_generation_prompt=True)
<|user|>
What is the Young's modulus of graphene?<|assistant|>
```

`add_generation_prompt=True`

is what appends that trailing `<|assistant|>\n`

. Without it the last token the model sees is `?`

, and it will cheerfully continue the *user's* turn, inventing a follow-up question instead of answering. With it, the model is standing exactly where, during training, the next token was the start of an answer.

Line the two up:

```
training:   <|user|>\nWhat is ... graphene?<|assistant|>\n | About 1 TPa.<|endoftext|>
inference:  <|user|>\nWhat is ... graphene?<|assistant|>\n | ← generation starts here
            └──────────── must match exactly ───────────┘
```

Everything left of the bar is the model's conditioning context. Training taught it "given this prefix, emit `About`

". If inference rebuilds that prefix with one extra newline, or a space the training path didn't have, the model is conditioning on something outside its training distribution.

What makes it nasty is that it degrades invisibly. Print both strings and they look identical. Nothing in the logs.

So just assert:

```
full   = tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=False)
prefix = tok.apply_chat_template(msgs[:-1], tokenize=True, add_generation_prompt=True)
assert full[:len(prefix)] == prefix
```

Three lines, and it hands you the loss-mask boundary for free: `len(prefix)`

is where the answer starts.

Roles are tokens, not fields, so there are no field boundaries to slice on. You locate the assistant span yourself.

The direct approach is the length difference:

```
labels = [-100] * len(prefix) + full[len(prefix):]
```

The boundary lands naturally after `<|assistant|>\n`

. Clean.

The cleverer-looking alternative is to regex the rendered text for `<|assistant|>`

and map character offsets back to token indices. Don't. The moment a message's `content`

legitimately contains that string it's wrong, and wrong silently.

The length-diff trick has one real limit: it only handles the last turn. For multi-turn data where you want loss on every assistant reply, you'd render incrementally turn by turn, which gets ugly fast.

The proper fix is to mark the assistant span in the template itself:

```
{%- if message['role'] == 'assistant' %}
    {% generation %}{{- message['content'] + '<|endoftext|>' }}{% endgeneration %}
{%- endif %}
```

Then ask for the mask:

```
out = tok.apply_chat_template(
    msgs, tokenize=True, return_dict=True,
    return_assistant_tokens_mask=True,
)
mask = out["assistant_masks"]   # 1 on assistant tokens
```

Correct for any number of turns. The cost is that the template has to contain `{% generation %}`

blocks, and a lot of published templates don't, which is why I haven't switched to it yet. Check first, or add the blocks to your own copy.

**Double BOS.** The template already emitted BOS. Call `tok(text)`

after it and the default `add_special_tokens=True`

adds another:

```
text = tok.apply_chat_template(msgs, tokenize=False)
ids  = tok(text, add_special_tokens=False)["input_ids"]   # ← required
```

Or skip the round trip with `tokenize=True`

, which uses `add_special_tokens=False`

internally anyway.

**Hand-built strings.** Writing `f"<|user|>\n{q}<|assistant|>\n"`

in your data pipeline works, right up until the official template changes, or you swap base models, or someone adds a system prompt. Let the template be the single source of truth and stop thinking about it.

**Inconsistent template branches.** Some hand-edited templates emit `<|assistant|>\n`

inside the loop and `<|assistant|>`

in the `add_generation_prompt`

branch, one newline short. The official GLM and Qwen templates are fine, but if you've touched a template, or added tool definitions or a system prompt, that assert is the only thing that will tell you.

**Special tokens in user content.** Jinja does plain string concatenation with no escaping. A `content`

field containing the literal text `<|assistant|>`

tokenizes into the real special token, and the model reads a genuine turn boundary. Prompt injection, training-data edition. If your corpus is scraped or model-generated, scan it first.

It comes down to one thing: the byte string you train on and the byte string you infer on have to share an identical prefix. Jinja2 produces that string, `tokenizer_config.json`

and `chat_template.jinja`

hold the recipe, and `add_generation_prompt`

is the only difference the two paths should have.

Honestly, none of this comes up most of the time. Use a stock model on a standard pipeline and `apply_chat_template`

handles everything; you never need to know what's underneath. It only surfaces when you start building your own data, computing your own masks, or swapping base models. And when it does surface it doesn't throw. It just makes things slightly worse, which is exactly why it's worth knowing in advance.

There's an isomorphic problem on the serving side. The QPS-as-load-signal mistake in [Little's Law and vLLM autoscaling](https://jessie-jia.com/article/littles-law-vllm-autoscaling/2026/08/07/) has the same shape: a metric that looks reasonable and is quietly measuring something else.

Further reading: the [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating) in the `transformers`

docs, and the whitespace-control section of the [Jinja2 template designer reference](https://jinja.palletsprojects.com/en/stable/templates/).