{"slug": "multi-token-prediction-done-right", "title": "Multi-Token Prediction Done Right", "summary": "A paper submitted to arXiv on 28 Oct 2026 identifies a flaw in standard Multi-Token Prediction (MTP) training, where a low signal-to-noise residual in the MTP cross-entropy gradient interferes with next-token and multi-token learning increasingly with scale. The authors introduce Soft-MTP, a one-line code change built on a KL gradient-recovery trick that backpropagates only the KL self-distillation signal by reusing existing NTP and MTP cross-entropy computations, adding no parameters and no measurable overhead. Across 167M to 1.2B parameters, Soft-MTP improves on standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, and stronger math and code reasoning after fine-tuning at 1.2B.", "body_md": "# Computer Science > Machine Learning\n\n  [Submitted on 28 Oct 2026]\n\n# Title:Multi-Token Prediction Done Right\n\n[View PDF](https://arxiv.org/pdf/2610.38331)\n\n[HTML (experimental)](https://arxiv.org/html/2610.38331v1)\n\nAbstract:Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing. Unlike minimizing the KL directly, this trick requires no vocabulary-sized logit matrices and is compatible with standard fused cross-entropy kernels. This enables Soft Multi-Token Prediction (Soft-MTP), a one-line code change to MTP that removes the noisy residual and backpropagates only the KL self-distillation signal by reusing the existing NTP and MTP cross-entropy computations. Soft-MTP supports both DeepSeek- and Meta-style MTP, adds no parameters, and incurs no measurable overhead. From 167M to 1.2B parameters, Soft-MTP improves over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. Soft-MTP also matches, if not exceeds, direct KL divergence minimization on most metrics. Soft-MTP is, in effect, a free lunch for MTP optimization. Our code can be found at [this https URL](https://github.com/JaydenTeoh/Soft-MTP).\n    \n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/multi-token-prediction-done-right", "canonical_source": "https://arxiv.org/abs/2610.38331", "published_at": "2026-11-04 06:46:01+00:00", "updated_at": "2026-11-04 07:20:25.276342+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["arXiv", "Soft-MTP", "Multi-Token Prediction", "DeepSeek", "Meta"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/multi-token-prediction-done-right", "markdown": "https://wpnews.pro/news/multi-token-prediction-done-right.md", "text": "https://wpnews.pro/news/multi-token-prediction-done-right.txt", "jsonld": "https://wpnews.pro/news/multi-token-prediction-done-right.jsonld"}}