What Actually Gets Tokenized in SFT

A developer dissects the tokenization process in supervised fine-tuning (SFT), revealing that chat templates are not metadata but actual tokens in the model's vocabulary. The analysis shows that roles like <|user|> and <|assistant|> are token IDs, and that Jinja2 templates control the exact string format, with whitespace stripping critical to matching training data. The developer warns against hardcoding token IDs across models and advises using repr() to verify template output.

tok.apply chat template msgs, add generation prompt=True I have copied that line more times than I can count without thinking about what any of it does. The official example writes it that way, the output looks right, you move on. Then you have to build your own SFT data and compute your own loss mask, and it turns out every piece of that line is load-bearing. I spent today pulling it apart. Notes below. The counterintuitive part first: the model has no idea messages exists. messages is for Python. Roles are real tokens in the vocabulary. tokenizer config.json or as a standalone chat template.jinja . When both exist the file wins. add generation prompt=False . Inference renders msgs :-1 with add generation prompt=True . messages exists I had some vague picture in my head where the role was structured metadata and something inside the model read it. Nope. {"role": "user", "content": "What is the Young's modulus of graphene?"} That list lives in your Python process and nowhere else. What reaches the model is: gMASK <sop <|user| What is the Young's modulus of graphene?<|assistant| and after tokenization: 151331, 151333, 151336, 198, ...content ids..., 151337, 198 Those leading numbers are GLM-4's ids for gMASK , <sop , <|user| and <|assistant| . Every model family has its own, so don't hardcode them across models. The role is a token, not a field. The model knows it is its turn to talk because it sees id 151337 , the same way it learns any other pattern. No JSON at inference time, no key lookup, no schema. One sequence. Nearly everything below follows from that. Jinja2 https://jinja.palletsprojects.com/ is a text templating engine originally built to render HTML for Flask. Three constructs, that's it: | Syntax | Purpose | |---|---| {{ expr }} | print the value of an expression | {% ... %} | control flow: for , if , set | { ... } | comment, produces no output | HuggingFace borrowed it for one job: flatten structured messages into the string the model was trained on, and flatten it the same way every time. The minimal shape: {%- for message in messages %} {{- '<|' + message 'role' + '| \n' + message 'content' + '<|endoftext| \n' }} {%- endfor %} {%- if add generation prompt %} {{- '<|assistant| \n' }} {%- endif %} {%- and -%} strip whitespace around the tag. Real templates are covered in them. I assumed this was a formatting preference. It isn't. Templates get indented so humans can read them, and without stripping, every newline and indent in the source lands verbatim in the prompt. One extra \n and the token sequence no longer matches what training saw. If you write your own, print the result with repr rather than trusting your eyes. transformers renders in a jinja2.sandbox.ImmutableSandboxedEnvironment , so arbitrary attribute access and side effects are blocked. You can't call random Python from in there. What you get: messages , plus tools when tool definitions are passed add generation prompt bos token , eos token and friends, from the tokenizer raise exception , for templates that reject malformed conversations, like a system message in the wrong slot strftime now , for templates that inject the current dateTwo places, same string. The old way is inline in tokenizer config.json . That's the file AutoTokenizer.from pretrained reads, and note it does not hold the vocabulary. That's in tokenizer.json , or vocab.json plus merges.txt , or a SentencePiece .model . It only records how to construct the tokenizer object: { "tokenizer class": "PreTrainedTokenizerFast", "model max length": 131072, "padding side": "left", "bos token": " gMASK ", "eos token": "<|endoftext| ", "pad token": "<|endoftext| ", "clean up tokenization spaces": false, "added tokens decoder": { "151329": { "content": "<|endoftext| ", "special": true, "normalized": false }, "151336": { "content": "<|user| ", "special": true, "normalized": false } }, "chat template": "{%- for message in messages %}..." // ← inline, one giant line } For SFT work, a few fields are worth a look. added tokens decoder decides whether <|user| is one atomic token or gets shredded into BPE pieces. Shredded, the role marker stops being a clean signal and the model has to infer the boundary from " < then | then user ". It can learn that. No reason to spend the capacity. padding side is right for training and must be left for batched generation. In a right-padded generation batch every sequence shorter than the longest one comes out garbage. eos token and pad token govern both where generation stops and which positions get carved out of the loss. The new way is a standalone chat template.jinja , which newer transformers versions write by default from save pretrained . The reason is mundane: inside JSON every newline is \n and every quote is escaped, so a 200-line template becomes one unreadable line with no diff and no highlighting. The file wins on precedence. Plenty of repos ship both for backward compatibility, which leaves a trap. Once the two drift apart, behavior depends on your transformers version, and that is a miserable thing to track down. Check they agree before you debug anything else. This is where I actually got stuck today. One SFT sample: msgs = {"role": "user", "content": "What is the Young's modulus of graphene?"}, {"role": "assistant", "content": "About 1 TPa."}, At training time you have the answer. That's the supervision signal, so you render the whole thing with no generation prompt: tok.apply chat template msgs, add generation prompt=False <|user| What is the Young's modulus of graphene?<|assistant| About 1 TPa.<|endoftext| Setting it to True would append a second, empty <|assistant| \n after the answer. A dangling role marker, which the model would learn to emit. At inference you don't have the answer. Producing it is the point, so you pass msgs :-1 : tok.apply chat template msgs :-1 , add generation prompt=True <|user| What is the Young's modulus of graphene?<|assistant| add generation prompt=True is what appends that trailing <|assistant| \n . Without it the last token the model sees is ? , and it will cheerfully continue the user's turn, inventing a follow-up question instead of answering. With it, the model is standing exactly where, during training, the next token was the start of an answer. Line the two up: training: <|user| \nWhat is ... graphene?<|assistant| \n | About 1 TPa.<|endoftext| inference: <|user| \nWhat is ... graphene?<|assistant| \n | ← generation starts here └──────────── must match exactly ───────────┘ Everything left of the bar is the model's conditioning context. Training taught it "given this prefix, emit About ". If inference rebuilds that prefix with one extra newline, or a space the training path didn't have, the model is conditioning on something outside its training distribution. What makes it nasty is that it degrades invisibly. Print both strings and they look identical. Nothing in the logs. So just assert: full = tok.apply chat template msgs, tokenize=True, add generation prompt=False prefix = tok.apply chat template msgs :-1 , tokenize=True, add generation prompt=True assert full :len prefix == prefix Three lines, and it hands you the loss-mask boundary for free: len prefix is where the answer starts. Roles are tokens, not fields, so there are no field boundaries to slice on. You locate the assistant span yourself. The direct approach is the length difference: labels = -100 len prefix + full len prefix : The boundary lands naturally after <|assistant| \n . Clean. The cleverer-looking alternative is to regex the rendered text for <|assistant| and map character offsets back to token indices. Don't. The moment a message's content legitimately contains that string it's wrong, and wrong silently. The length-diff trick has one real limit: it only handles the last turn. For multi-turn data where you want loss on every assistant reply, you'd render incrementally turn by turn, which gets ugly fast. The proper fix is to mark the assistant span in the template itself: {%- if message 'role' == 'assistant' %} {% generation %}{{- message 'content' + '<|endoftext| ' }}{% endgeneration %} {%- endif %} Then ask for the mask: out = tok.apply chat template msgs, tokenize=True, return dict=True, return assistant tokens mask=True, mask = out "assistant masks" 1 on assistant tokens Correct for any number of turns. The cost is that the template has to contain {% generation %} blocks, and a lot of published templates don't, which is why I haven't switched to it yet. Check first, or add the blocks to your own copy. Double BOS. The template already emitted BOS. Call tok text after it and the default add special tokens=True adds another: text = tok.apply chat template msgs, tokenize=False ids = tok text, add special tokens=False "input ids" ← required Or skip the round trip with tokenize=True , which uses add special tokens=False internally anyway. Hand-built strings. Writing f"<|user| \n{q}<|assistant| \n" in your data pipeline works, right up until the official template changes, or you swap base models, or someone adds a system prompt. Let the template be the single source of truth and stop thinking about it. Inconsistent template branches. Some hand-edited templates emit <|assistant| \n inside the loop and <|assistant| in the add generation prompt branch, one newline short. The official GLM and Qwen templates are fine, but if you've touched a template, or added tool definitions or a system prompt, that assert is the only thing that will tell you. Special tokens in user content. Jinja does plain string concatenation with no escaping. A content field containing the literal text <|assistant| tokenizes into the real special token, and the model reads a genuine turn boundary. Prompt injection, training-data edition. If your corpus is scraped or model-generated, scan it first. It comes down to one thing: the byte string you train on and the byte string you infer on have to share an identical prefix. Jinja2 produces that string, tokenizer config.json and chat template.jinja hold the recipe, and add generation prompt is the only difference the two paths should have. Honestly, none of this comes up most of the time. Use a stock model on a standard pipeline and apply chat template handles everything; you never need to know what's underneath. It only surfaces when you start building your own data, computing your own masks, or swapping base models. And when it does surface it doesn't throw. It just makes things slightly worse, which is exactly why it's worth knowing in advance. There's an isomorphic problem on the serving side. The QPS-as-load-signal mistake in Little's Law and vLLM autoscaling https://jessie-jia.com/article/littles-law-vllm-autoscaling/2026/08/07/ has the same shape: a metric that looks reasonable and is quietly measuring something else. Further reading: the chat templating guide https://huggingface.co/docs/transformers/main/en/chat templating in the transformers docs, and the whitespace-control section of the Jinja2 template designer reference https://jinja.palletsprojects.com/en/stable/templates/ .