What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language Researchers introduced tree-grams, autoregressive generative n-gram models that build the conditional probability tensor p(x_n | x_1:n-1) through hierarchical composition of order-2 and order-3 tensors, in arXiv paper 2611.23093v1. Tree-grams outperform existing n-gram baselines while remaining competitive with deep transformer-based language models, and the authors report that routing across candidate topologies produces emergent sparsity allowing pruning of up to 95% of topologies without significant performance change. The work also shows hyperparameter transfer under μP becomes more robust when narrower models are tuned on simpler data generated by tree-grams with moderate n and d, with entropy and effective correlation range controlling the performance gap after fine-tuning. arXiv:2611.23093v1 Announce Type: new Abstract: Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p x n \mid x {1:n-1} $ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $\mu$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.