# What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

> Source: <https://www.machinebrief.com/news/whats-in-a-tree-hierarchical-dollarndollar-gram-models-as-co-1hzj>
> Published: 2026-11-19 05:00:00+00:00

**Recommendation:** When tuning narrower models under μP, pretrain on simpler synthetic data from tree-grams with moderate n and d to make hyperparameter transfer more robust, then prune up to 95% of routed topologies with no significant performance loss.
More in the playbook "Fine-tuning and data": <https://wpnews.pro/playbooks/training-data>

arXiv:2611.23093v1 Announce Type: new 
Abstract: Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $\mu$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.
