Hi everyone,
I am sharing recent results from my independent research project, LIMEN (Liminal Internal Metric for Emergent Navigation), which aims to characterize the internal dynamics of Transformers through hidden state analysis.
Following our previous findings that functional information is encoded in the relative geometry of representations rather than individual neurons (V22), this new phase focuses on the impact of context (ambiguity) and the temporal structure of state transitions (V23–V24).
Context & Methodology
Model Panel: 7 open-source models (GPT-2, DistilGPT2, OPT-125M, Qwen2.5-0.5B, TinyLlama-1.1B, Phi-1.5, Llama-3.2-1B). Approach: Layer-by-layer analysis of latent trajectories, linear probe decoding, and symbolic analysis of dynamic regimes.
Philosophy: Strict empiricism. Clear distinction between observation, interpretation, and speculation. Code and data are available upon request.
V23: The Impact of Ambiguity on Internal Dynamics
The objective was to determine whether semantic ambiguity alters the model’s “cognitive trajectory.”
Key Findings (V23.2b): AMBIGUITY_AFFECTS_TRAJECTORY = YES: Ambiguity significantly modifies trajectory geometry (curvature, cosine similarity).
AMBIGUITY_INCREASES_INSTABILITY = NO: Counter-intuitively, ambiguity does not increase global chaos. Instead, the model becomes geometrically more “cautious.”
AMBIGUITY_DELAYS_COMMITMENT = PARTIAL: Modern models (Phi-1.5, Llama-3.2) delay their decisional engagement when facing uncertainty, spending more time in exploration regimes.
Architectural Signature: Phi-1.5 shows unique sensitivity, increasing its occupancy of the bifurcation regime (D_STATE) under ambiguity, suggesting a distinct iterative reasoning mechanism compared to standard completion models.
Related Preprint: Conditional Dynamic Signatures in Large Language Models
V24: Discovery of a “Universal Dynamic Grammar”
By shifting from continuous analysis to a symbolic analysis of state sequences, a striking structure emerged.
Key Findings (V24.1): STATE_GRAMMAR_EXISTS = YES: Trajectories are not random. They follow strict transitional patterns.
UNIVERSAL_GRAMMAR = YES: Seven transition motifs are conserved across all tested architectures, notably:
B→B (Initial Hesitation/Exploration) B→A (Convergence toward stable processing)
A→A (Maintenance of the adaptive regime – the primary attractor)
A→D (Transition to final decision)
Funnel Structure: Typical dynamics follow an Exploration (B) → Stabilization/Processing (A) → Decision (D) schema. State A acts as a strong attractor (
𝑃
( 𝐴
→
𝐴
) ≈
0.91
P(A→A)≈0.91). The Phi-1.5 Exception: Unlike other models that quickly converge to A, Phi-1.5 maintains complex B↔A oscillations throughout the depth, confirming its nature as a “reasoning” model rather than a simple statistical completer.
Related Preprint: A Runtime Trajectory Dynamics Framework for Large Language Models (updated)
Implications & Discussion
These results suggest that Transformer “intelligence” is not just a matter of static weights, but of constrained geometric navigation.
Auditability: A violation of this universal grammar (e.g., a direct B→D jump without an A phase) could be an early indicator of hallucination or reasoning errors.
Control: Understanding these attractors opens the door to more precise dynamic steering than prompt engineering alone.
Open Questions for the Community:
Have you observed violations of this B→A→D grammar in cases of blatant hallucinations?
How do these motifs evolve in very large models (>70B) where depth is significantly greater?
Are there recent publications on the “symbolic dynamics” of hidden states that align with these findings?
I welcome any methodological criticism, suggestions for additional controls, or collaboration.
Best regards,