cd /news/large-language-models/deep-latents-long-memory-latent-pers… · home › topics › large-language-models › article
[ARTICLE · art-170376] src=machinebrief.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Deep Latents, Long Memory: Latent persistence explains temporal correlations in transformer activations

A paper posted as arXiv:2611.18089v1 proposes measuring a transformer layer's autocorrelation — the average dot product between activations at two context positions separated by a lag — and reports that slowly decaying tails grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, the growth stems from encoding abstract latent variables that govern long stretches of input, and in real text the persistence of sparse-autoencoder features correlates strongly with autocorrelation magnitude across datasets and LLMs, with persistence time increasing with depth. Ablating the persistent sparse-autoencoder features produced a corresponding decrease in autocorrelations.

by read1 min views1 publishedNov 16, 2026

arXiv:2611.18089v1 Announce Type: cross Abstract: The hidden layers of Large Language Models (LLMs) are believed to encode representations of the data whose abstraction increases with depth. However, in the absence of an operational notion of abstraction, this statement is hard to test quantitatively. Here, we propose an alternative view, building on the principle that the correlations of a sequence reflect the structure of the process that generated it. We define the autocorrelation function of a layer as the average dot product between its activations at two context positions separated by a lag. Latent variables encoded persistently along the context create slowly decaying tails in the autocorrelation, and we find these tails to grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, this growth stems from the encoding of abstract latent variables, which govern long stretches of the input. Indeed, the persistence of latent codes predicts the qualitative structure of autocorrelations and accounts for most of the empirical measures. In real text, we take sparse-autoencoder features as proxies for latent variables, and find that their persistence correlates strongly with the magnitude of autocorrelations across datasets and LLMs. In particular, the persistence time increases with depth, as do autocorrelations, implying that deeper latent features persist for longer context spans. We complement this analysis by explicitly ablating the persistent sparse-autoencoder features and observing a consequent decrease in the autocorrelations.

── more in #large-language-models 4 stories · sorted by recency
── more on @arxiv 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/deep-latents-long-me…] indexed:0 read:1min 2026-11-16 · —