High-dimensional layer-state trajectories reveal the internal dynamics of transformer inference

A study posted as arXiv:2610.26866v1 reports that layer order in transformer inference defines a measurable computation: across representative Qwen, Llama and Gemma checkpoints, realized layer order produced hidden-state paths 1.60–2.24 times as aligned with their endpoint chord as endpoint-preserving shuffles. The order effect transferred to a lexical task and was stronger in three random initializations of the same Qwen architecture, which the authors say identifies an architecture-compatible scaffold rather than a training-specific law. A confidence gate drove 17 of 87 held-out Qwen closure trajectories across the full-vocabulary top-1 candidate boundary while leaving all 203 non-closure top-1 outputs unchanged, though the authors state the work does not establish a complete theory of LLM dynamics or control of answer correctness.

arXiv:2610.26866v1 Announce Type: new Abstract: Mechanistic studies of large language models localize function in parameters, attention heads or neurons, but those parts do not specify how one inference unfolds. We treat the ordered layer-state trajectory as the empirical process and test whether layer order defines a measurable computation. Across representative Qwen, Llama and Gemma checkpoints, realized order produced paths 1.60--2.24 times as aligned with their endpoint chord as endpoint-preserving shuffles. The order effect transferred to a lexical task, but was stronger in three random initializations of the same Qwen architecture, identifying an architecture-compatible scaffold rather than a training-specific law. Direct hidden states provided stronger mechanism coordinates than vocabulary readbacks, and a task-defined coordinate made candidate competition computable. After additive interventions failed, local transition actuators selectively controlled internal trajectories across checkpoints. A confidence gate then drove 17 of 87 held-out Qwen closure trajectories across the full-vocabulary top-1 candidate boundary while leaving all 203 non-closure top-1 outputs unchanged. Safety-matched action reassignment performed similarly, locating the gain in selective gating and the bounded actuator family rather than fine-grained dose assignment. Layer order exposes an observable, computable and behaviourally consequential process of transformer inference without establishing a complete theory of LLM dynamics or control of answer correctness.