# Do Transformer representations progressively structure across depth and time? Results from 8 open models

> Source: <https://discuss.huggingface.co/t/do-transformer-representations-progressively-structure-across-depth-and-time-results-from-8-open-models/179289#post_1>
> Published: 2026-08-26 18:52:16+00:00

Hi everyone,

I’ve just published a new preprint that brings together several months of experiments on hidden-state dynamics in small open Transformer models.

The question is fairly simple:

During inference, do internal representations simply change from layer to layer, or is there evidence of a more structured progression across depth and generation time?

I tried to study this without assuming that hidden-state dynamics are equivalent to “reasoning”.

The working framework is:

tokens → embeddings → contextualisation → relational structuring → functional structuring → decision formation → projection

This is a descriptive hypothesis about representation dynamics, not a claim that these stages correspond to a universal reasoning mechanism.

The expanded study uses 8 locally instrumented open models, with synchronized hidden-state and output observations and explicit separation between:

depth — what changes as information passes through Transformer layers

time — what changes as autoregressive generation progresses

A few results were particularly interesting.

First, local ordering across model depth survived expansion.

The observed ordering was significantly more structured than random layer permutations (`p = 0.00019996`

) and remained supported when each model was removed from the panel one at a time (**8/8 leave-one-model-out checks**).

Second, cross-model depth profiles remained surprisingly coherent.

The mean correlation across normalized depth profiles was approximately r = 0.789.

This does not mean that all models follow the same trajectory. Rather, it suggests that some aspects of where changes occur along depth may be more shared than I initially expected.

Third, functionally labelled events were not uniformly distributed across depth.

Event type showed a statistically supported association with normalized layer depth (`p = 0.0024`

).

I’m deliberately calling this an association, not evidence of a causal mechanism.

But one of the most useful results was actually a failure to replicate.

In an earlier smaller panel, a common temporal pattern in local trajectory instability looked promising. After expanding the panel, that common temporal mode disappeared — it survived 0/8 leave-one-model-out checks.

Two other intuitive hypotheses also failed:

models with similar observed functional outcomes were not significantly more structurally similar (`p = 0.408`

), and models from the same architecture family were not significantly more similar either (`p = 0.771`

).

To me, this is probably the most important part of the result.

The data do not support a simple story where architecture determines one characteristic trajectory or where one universal temporal dynamic explains inference.

What remains is a narrower hypothesis:

Transformer inference may contain reproducible structure along depth while remaining highly conditional in time and behavior.

I refer to this as Progressive Representational Structuring.

The framework is summarized by:

Representation ≠ Function ≠ Behavior

A representation can contain information without that information yet serving the same function, and a functional transition does not guarantee a particular final behavior.

I would be especially interested in feedback from people working on:

mechanistic interpretability, activation patching, probing, hidden-state geometry, steering, representation engineering, or larger open models.

In particular, I’m curious whether others observe similar **ordered depth structure without a universal temporal trajectory.

Preprint:

**Progressive Representational Structuring in Small Language Models: Functionally Labelled Trajectories Across Depth and Time**

DOI: 10.5281/zenodo.22116637

This is still descriptive work. Causal intervention and structural-transfer experiments are separate next steps rather than claims of this paper. [Progressive Representational Structuring in Small Language Models: Functionally Labelled Trajectories Across Depth and Time | Zenodo](https://doi.org/10.5281/zenodo.22116637)