{"slug": "model-can-generalize-code-better-with-programming-language-aware-tokenization", "title": "Model can generalize code better with programming-language-aware tokenization", "summary": "Extending the vocabulary of Qwen3.5-4B-Base with missing Python, C++, Go and Rust keywords and fine-tuning on a SWE dataset raised HumanEval accuracy from 45.7% to 52.4% and HumanEval+ from 39.0% to 47.0%, according to the developer's published experiment, code, model weights and checkpoints. The same recipe applied to the base model with its own tokenizer served as the comparison, and MBPP on Evalplus slipped slightly from 65.3% to 64.6%. The developer reports that tokenizers in Nvidia Nemotron 3 and Qwen 3.5 can split operators such as \"<<=\" into separate tokens, forcing models to relearn that the pieces form one lexical unit.", "body_md": "I have been developing a SWE model with my own code, chats, conversations for my personal agent. While tinkering with Nvidia Nemotron 3 and Qwen 3.5, I found that while their tokenizer preserves most of the operators and delimiters, they can sometimes break keywords, operators of the language in awkward places. For example it can break “<<=” operator to multiple different tokens as “<<”, “=” or “<”, “<=”.\n\nProgramming language has its own syntax, semantics, structure and “grammar” where the keywords, operators carry semantic meaning. A basic unit is called Lexical token which is recognized by the language lexer.\n\nWhen lexical token is split to multiple tokens or merged together the meaning or behavior can change entirely, now the model has to learn that those separate pieces form a single unit or the merged vocab actually have separate meaning.\n\nSo to preserve the lexical token boundaries, I extended the vocab of Qwen3.5-4B-Base model with missing keywords from python, c++, golang, rust etc.. And finetuned on SWE dataset, then I finetuned the base model with its own tokenizer with the same recipe to compare the performance.\n\nThe extended-vocabulary model improved on HumanEval from 45.7% to 52.4% and HumanEval+ from 39.0% to 47.0%. MBPP scores slightly decreased from 65.3% to 64.6% on Evalplus.\n\nI have published about the experiment, code, model weights and checkpoints here", "url": "https://wpnews.pro/news/model-can-generalize-code-better-with-programming-language-aware-tokenization", "canonical_source": "https://discuss.huggingface.co/t/model-can-generalize-code-better-with-programming-language-aware-tokenization/190957#post_1", "published_at": "2026-10-23 17:51:11+00:00", "updated_at": "2026-10-23 18:10:15.433361+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["Qwen3.5-4B-Base", "Nvidia Nemotron 3", "Qwen 3.5", "HumanEval", "HumanEval+", "MBPP", "Evalplus"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/model-can-generalize-code-better-with-programming-language-aware-tokenization", "markdown": "https://wpnews.pro/news/model-can-generalize-code-better-with-programming-language-aware-tokenization.md", "text": "https://wpnews.pro/news/model-can-generalize-code-better-with-programming-language-aware-tokenization.txt", "jsonld": "https://wpnews.pro/news/model-can-generalize-code-better-with-programming-language-aware-tokenization.jsonld"}}