{"slug": "exploring-functional-regimes-inside-small-language-models-independent-research", "title": "Exploring Functional Regimes Inside Small Language Models (Independent Research)", "summary": "An independent research project analyzed the internal dynamics of small and medium-sized language models, revealing that functional properties become linearly decodable in hidden representations and that models cluster into two behavioral groups based on dynamic and functional profiles. The findings suggest that functional signal depends more on the geometry of representation space than on specific dimensions, aligning with mechanistic interpretability ideas.", "body_md": "Hi everyone,\n\nOver the last few months I’ve been working on an independent research project exploring the **internal dynamics** of small and medium-sized language models.\n\nRather than evaluating models only by their outputs (benchmarks, perplexity, etc.), I’m trying to characterize **how their hidden representations evolve during inference**.\n\nThe project currently covers 7 open models:\n\nThe first part of the framework studies internal trajectories through hidden-state dynamics.\n\nInstead of asking *“Which model is more accurate?”*, I ask:\n\nThis produced several reproducible dynamical fingerprints and architecture clusters.\n\nThe second phase moves away from pure dynamics and investigates whether different functional properties become linearly decodable inside hidden representations.\n\nAcross multiple probe experiments I observed evidence that:\n\nOne interesting observation is that the position of these high-capacity regions varies across architectures rather than appearing at identical absolute depths.\n\nThe result that surprised me the most came from a series of control experiments.\n\nAfter training linear probes I compared:\n\nGaussian noise and feature permutation substantially reduced decodability.\n\nOrthogonal rotations, however, preserved it almost entirely.\n\nThat suggests (at least empirically) that the functional signal depends more on the **geometry of the representation space** than on specific embedding dimensions.\n\nThis seems broadly consistent with ideas discussed in mechanistic interpretability about distributed feature directions.\n\nAcross several independent audits, the models repeatedly separate into two broad behavioral groups.\n\n**Cluster A**\n\nThese models consistently exhibit similar dynamic and functional profiles.\n\n**Cluster B**\n\nDespite architectural differences, these models repeatedly cluster together across multiple analyses.\n\nSeeing the same grouping emerge from different metrics was one of the motivations for continuing the project.\n\nI’m now moving from observation toward causal testing.\n\nThe next experiments aim to answer questions such as:\n\nThis is entirely independent research, so I’d genuinely appreciate feedback.\n\nI’m especially interested in hearing from people working on:\n\nI’d love to know whether these observations resonate with existing work—or whether there are obvious control experiments I should run next.", "url": "https://wpnews.pro/news/exploring-functional-regimes-inside-small-language-models-independent-research", "canonical_source": "https://discuss.huggingface.co/t/exploring-functional-regimes-inside-small-language-models-independent-research/177219#post_1", "published_at": "2026-06-29 02:23:53+00:00", "updated_at": "2026-06-29 02:36:38.931046+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "neural-networks"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/exploring-functional-regimes-inside-small-language-models-independent-research", "markdown": "https://wpnews.pro/news/exploring-functional-regimes-inside-small-language-models-independent-research.md", "text": "https://wpnews.pro/news/exploring-functional-regimes-inside-small-language-models-independent-research.txt", "jsonld": "https://wpnews.pro/news/exploring-functional-regimes-inside-small-language-models-independent-research.jsonld"}}