[Submitted on 8 Nov 2025 (
[v1](https://arxiv.org/abs/2511.05963v1)), last revised 15 Jun 2026 (this version, v4)]# Title:Next-Latent Prediction Transformers Learn Compact World Models
[View PDF](/pdf/2511.05963)
[HTML (experimental)](https://arxiv.org/html/2511.05963v4)
Abstract:Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens. Consequently, they lack an inherent incentive to compress history into compact latent states with consistent transition rules. This often leads to learning solutions that generalize poorly. We introduce Next-Latent Prediction (NextLat), which extends standard next-token training with self-supervised predictions in the latent space. Specifically, NextLat trains a transformer to learn latent representations that are predictive of its next latent state given the next token. Theoretically, we show that these latents provably converge towards belief states, compressed information about the history necessary to predict the future. This simple auxiliary objective injects a recurrent inductive bias into transformers while leaving their architecture, parallel training efficiency, and inference unchanged. NextLat effectively encourages transformers to form compact internal world models with coherent belief states and transition dynamics -- crucial properties not guaranteed by standard next-token prediction alone. Empirically, across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat demonstrates significant gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning. Furthermore, NextLat enables variable-length self-speculative decoding, accelerating inference by up to 3.3x in language modeling. NextLat offers a simple yet effective paradigm for learning compact, predictive representations in transformers that generalize better. Our code is available at[this https URL].
Submission history #
From: Jayden Teoh [[view email](/show-email/f90ec165/2511.05963)]
**Sat, 8 Nov 2025 10:41:26 UTC (7,302 KB)**
[[v1]](/abs/2511.05963v1)**Fri, 22 May 2026 06:33:12 UTC (9,033 KB)**
[[v2]](/abs/2511.05963v2)**Mon, 25 May 2026 15:53:24 UTC (9,038 KB)**
[[v3]](/abs/2511.05963v3)**[v4]** Mon, 15 Jun 2026 08:56:56 UTC (8,940 KB)
References & Citations
...
Bibliographic Explorer
(What is the Explorer?) Connected Papers
(What is Connected Papers?) Litmaps
(What is Litmaps?) scite Smart Citations
(What are Smart Citations?)# Code, Data and Media Associated with this Article alphaXiv
(What is alphaXiv?) CatalyzeX Code Finder for Papers
(What is CatalyzeX?) DagsHub
(What is DagsHub?) Gotit.pub
(What is GotitPub?) Hugging Face
(What is Huggingface?) ScienceCast
(What is ScienceCast?)# Demos Influence Flower
(What are Influence Flowers?) CORE Recommender
(What is CORE?) IArxiv Recommender
(What is IArxiv?)# arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.