# Latent Fact Composition under Next-Token Training

> Source: <https://www.machinebrief.com/news/latent-fact-composition-under-next-token-training-lyxm>
> Published: 2026-11-18 05:00:00+00:00

**Key takeaway:** Training a looped transformer to predict only the answer, with the intermediate entity passed to the second loop as a token embedding, yields near-perfect accuracy on held-out two-hop questions, but full next-token training causes accuracy to rise then decline in the second hop even while both facts

arXiv:2611.21921v1 Announce Type: new 
Abstract: Controlled studies of latent two-hop reasoning train transformers to predict only the answer, whereas language models are trained to predict every token. We ask whether a model trained on every token learns to answer two-hop questions whose intermediate entity appears in no two-hop training example. We build a synthetic corpus with such questions. On this corpus every term of the loss can be computed exactly. A looped transformer, a standard transformer and DiscoLoop answer almost none of these questions. We therefore introduce the Baton family of looped transformers. They pass the intermediate entity to the second loop as a token embedding. When trained to predict only the answer, two of its three variants answer these questions almost perfectly. We then relate the success and failure of each model to three design conditions. Experiments that change one condition at a time agree with this hypothesis. When Baton is trained on every token, however, its accuracy on these questions rises and then declines. We further split the gradient of the next-token loss into three terms. Removing the two terms that depend on the particular training set raises the peak accuracy but does not remove the decline. The decline arises in the second hop while the model still recalls both facts.
