{"slug": "latent-fact-composition-under-next-token-training", "title": "Latent Fact Composition under Next-Token Training", "summary": "A new arXiv paper (2611.21921v1) reports that a looped transformer design called Baton, which passes the intermediate entity to the second loop as a token embedding, answers nearly all synthetic two-hop questions whose intermediate entity appears in no two-hop training example when trained to predict only the answer — while a standard transformer, a looped transformer and DiscoLoop answer almost none. When Baton is trained on every token, as language models are, its accuracy on these questions rises and then declines, with the decline arising in the second hop while the model still recalls both facts; removing the two gradient terms that depend on the particular training set raised peak accuracy but did not remove the decline. The authors tie each model's success or failure to three design conditions, confirmed by experiments that change one condition at a time.", "body_md": "arXiv:2611.21921v1 Announce Type: new \nAbstract: Controlled studies of latent two-hop reasoning train transformers to predict only the answer, whereas language models are trained to predict every token. We ask whether a model trained on every token learns to answer two-hop questions whose intermediate entity appears in no two-hop training example. We build a synthetic corpus with such questions. On this corpus every term of the loss can be computed exactly. A looped transformer, a standard transformer and DiscoLoop answer almost none of these questions. We therefore introduce the Baton family of looped transformers. They pass the intermediate entity to the second loop as a token embedding. When trained to predict only the answer, two of its three variants answer these questions almost perfectly. We then relate the success and failure of each model to three design conditions. Experiments that change one condition at a time agree with this hypothesis. When Baton is trained on every token, however, its accuracy on these questions rises and then declines. We further split the gradient of the next-token loss into three terms. Removing the two terms that depend on the particular training set raises the peak accuracy but does not remove the decline. The decline arises in the second hop while the model still recalls both facts.", "url": "https://wpnews.pro/news/latent-fact-composition-under-next-token-training", "canonical_source": "https://www.machinebrief.com/news/latent-fact-composition-under-next-token-training-lyxm", "published_at": "2026-11-18 05:00:00+00:00", "updated_at": "2026-11-18 05:16:03.415171+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "neural-networks"], "entities": ["Baton", "DiscoLoop", "arXiv"], "also_reported_by": [], "takeaway": "Training a looped transformer to predict only the answer, with the intermediate entity passed to the second loop as a token embedding, yields near-perfect accuracy on held-out two-hop questions, but full next-token training causes accuracy to rise then decline in the second hop even while both facts", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/latent-fact-composition-under-next-token-training", "markdown": "https://wpnews.pro/news/latent-fact-composition-under-next-token-training.md", "text": "https://wpnews.pro/news/latent-fact-composition-under-next-token-training.txt", "jsonld": "https://wpnews.pro/news/latent-fact-composition-under-next-token-training.jsonld"}}