cd /news/artificial-intelligence/latent-fact-composition-under-next-t… · home › topics › artificial-intelligence › article
[ARTICLE · art-171560] src=machinebrief.com ↗ pub= topic=artificial-intelligence verified=true sentiment=· neutral

Latent Fact Composition under Next-Token Training

A new arXiv paper (2611.21921v1) reports that a looped transformer design called Baton, which passes the intermediate entity to the second loop as a token embedding, answers nearly all synthetic two-hop questions whose intermediate entity appears in no two-hop training example when trained to predict only the answer — while a standard transformer, a looped transformer and DiscoLoop answer almost none. When Baton is trained on every token, as language models are, its accuracy on these questions rises and then declines, with the decline arising in the second hop while the model still recalls both facts; removing the two gradient terms that depend on the particular training set raised peak accuracy but did not remove the decline. The authors tie each model's success or failure to three design conditions, confirmed by experiments that change one condition at a time.

key takeaway

Training a looped transformer to predict only the answer, with the intermediate entity passed to the second loop as a token embedding, yields near-perfect accuracy on held-out two-hop questions, but full next-token training causes accuracy to rise then decline in the second hop even while both facts

by read1 min views2 publishedNov 18, 2026

arXiv:2611.21921v1 Announce Type: new Abstract: Controlled studies of latent two-hop reasoning train transformers to predict only the answer, whereas language models are trained to predict every token. We ask whether a model trained on every token learns to answer two-hop questions whose intermediate entity appears in no two-hop training example. We build a synthetic corpus with such questions. On this corpus every term of the loss can be computed exactly. A looped transformer, a standard transformer and DiscoLoop answer almost none of these questions. We therefore introduce the Baton family of looped transformers. They pass the intermediate entity to the second loop as a token embedding. When trained to predict only the answer, two of its three variants answer these questions almost perfectly. We then relate the success and failure of each model to three design conditions. Experiments that change one condition at a time agree with this hypothesis. When Baton is trained on every token, however, its accuracy on these questions rises and then declines. We further split the gradient of the next-token loss into three terms. Removing the two terms that depend on the particular training set raises the peak accuracy but does not remove the decline. The decline arises in the second hop while the model still recalls both facts.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @baton 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/latent-fact-composit…] indexed:0 read:1min 2026-11-18 · —