I have been developing a SWE model with my own code, chats, conversations for my personal agent. While tinkering with Nvidia Nemotron 3 and Qwen 3.5, I found that while their tokenizer preserves most of the operators and delimiters, they can sometimes break keywords, operators of the language in awkward places. For example it can break “<<=” operator to multiple different tokens as “<<”, “=” or “<”, “<=”.
Programming language has its own syntax, semantics, structure and “grammar” where the keywords, operators carry semantic meaning. A basic unit is called Lexical token which is recognized by the language lexer.
When lexical token is split to multiple tokens or merged together the meaning or behavior can change entirely, now the model has to learn that those separate pieces form a single unit or the merged vocab actually have separate meaning.
So to preserve the lexical token boundaries, I extended the vocab of Qwen3.5-4B-Base model with missing keywords from python, c++, golang, rust etc.. And finetuned on SWE dataset, then I finetuned the base model with its own tokenizer with the same recipe to compare the performance.
The extended-vocabulary model improved on HumanEval from 45.7% to 52.4% and HumanEval+ from 39.0% to 47.0%. MBPP scores slightly decreased from 65.3% to 64.6% on Evalplus.
I have published about the experiment, code, model weights and checkpoints here