Next-Latent Prediction Transformers Learn Compact World Models (2025)

Researchers introduced Next-Latent Prediction (NextLat), a training method that extends next-token prediction with self-supervised latent state predictions, enabling transformers to learn compact world models with improved generalization. The method demonstrated gains in accuracy, representation compression, and inference speed (up to 3.3x) across benchmarks in world modeling, reasoning, planning, and language modeling.

Computer Science Machine Learning Submitted on 8 Nov 2025 v1 https://arxiv.org/abs/2511.05963v1 , last revised 15 Jun 2026 this version, v4 Title:Next-Latent Prediction Transformers Learn Compact World Models View PDF /pdf/2511.05963 HTML experimental https://arxiv.org/html/2511.05963v4 Abstract:Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens. Consequently, they lack an inherent incentive to compress history into compact latent states with consistent transition rules. This often leads to learning solutions that generalize poorly. We introduce Next-Latent Prediction NextLat , which extends standard next-token training with self-supervised predictions in the latent space. Specifically, NextLat trains a transformer to learn latent representations that are predictive of its next latent state given the next token. Theoretically, we show that these latents provably converge towards belief states, compressed information about the history necessary to predict the future. This simple auxiliary objective injects a recurrent inductive bias into transformers while leaving their architecture, parallel training efficiency, and inference unchanged. NextLat effectively encourages transformers to form compact internal world models with coherent belief states and transition dynamics -- crucial properties not guaranteed by standard next-token prediction alone. Empirically, across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat demonstrates significant gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning. Furthermore, NextLat enables variable-length self-speculative decoding, accelerating inference by up to 3.3x in language modeling. NextLat offers a simple yet effective paradigm for learning compact, predictive representations in transformers that generalize better. Our code is available at this https URL . Submission history From: Jayden Teoh view email /show-email/f90ec165/2511.05963 Sat, 8 Nov 2025 10:41:26 UTC 7,302 KB v1 /abs/2511.05963v1 Fri, 22 May 2026 06:33:12 UTC 9,033 KB v2 /abs/2511.05963v2 Mon, 25 May 2026 15:53:24 UTC 9,038 KB v3 /abs/2511.05963v3 v4 Mon, 15 Jun 2026 08:56:56 UTC 8,940 KB References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender IArxiv Recommender What is IArxiv? https://iarxiv.org/about arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .