Deep Latents, Long Memory: Latent persistence explains temporal correlations in transformer activations A paper posted as arXiv:2611.18089v1 proposes measuring a transformer layer's autocorrelation — the average dot product between activations at two context positions separated by a lag — and reports that slowly decaying tails grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, the growth stems from encoding abstract latent variables that govern long stretches of input, and in real text the persistence of sparse-autoencoder features correlates strongly with autocorrelation magnitude across datasets and LLMs, with persistence time increasing with depth. Ablating the persistent sparse-autoencoder features produced a corresponding decrease in autocorrelations. arXiv:2611.18089v1 Announce Type: cross Abstract: The hidden layers of Large Language Models LLMs are believed to encode representations of the data whose abstraction increases with depth. However, in the absence of an operational notion of abstraction, this statement is hard to test quantitatively. Here, we propose an alternative view, building on the principle that the correlations of a sequence reflect the structure of the process that generated it. We define the autocorrelation function of a layer as the average dot product between its activations at two context positions separated by a lag. Latent variables encoded persistently along the context create slowly decaying tails in the autocorrelation, and we find these tails to grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, this growth stems from the encoding of abstract latent variables, which govern long stretches of the input. Indeed, the persistence of latent codes predicts the qualitative structure of autocorrelations and accounts for most of the empirical measures. In real text, we take sparse-autoencoder features as proxies for latent variables, and find that their persistence correlates strongly with the magnitude of autocorrelations across datasets and LLMs. In particular, the persistence time increases with depth, as do autocorrelations, implying that deeper latent features persist for longer context spans. We complement this analysis by explicitly ablating the persistent sparse-autoencoder features and observing a consequent decrease in the autocorrelations.