{"slug": "what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of", "title": "What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language", "summary": "Researchers introduced tree-grams, autoregressive generative n-gram models that build the conditional probability tensor p(x_n | x_1:n-1) through hierarchical composition of order-2 and order-3 tensors, in arXiv paper 2611.23093v1. Tree-grams outperform existing n-gram baselines while remaining competitive with deep transformer-based language models, and the authors report that routing across candidate topologies produces emergent sparsity allowing pruning of up to 95% of topologies without significant performance change. The work also shows hyperparameter transfer under μP becomes more robust when narrower models are tuned on simpler data generated by tree-grams with moderate n and d, with entropy and effective correlation range controlling the performance gap after fine-tuning.", "body_md": "arXiv:2611.23093v1 Announce Type: new \nAbstract: Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \\mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $\\mu$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\\% of topologies without significant change in performance.", "url": "https://wpnews.pro/news/what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of", "canonical_source": "https://www.machinebrief.com/news/whats-in-a-tree-hierarchical-dollarndollar-gram-models-as-co-1hzj", "published_at": "2026-11-19 05:00:00+00:00", "updated_at": "2026-11-19 05:43:08.806599+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "natural-language-processing"], "entities": ["tree-grams", "μP", "arXiv"], "also_reported_by": [], "takeaway": "Tree-grams' concentrated routing lets you prune up to 95% of candidate topologies without significant performance change, and tuning narrower models on simpler tree-gram-generated data (moderate n and d) makes μP hyperparameter transfer more robust.", "recommendation": {"advice": "When tuning narrower models under μP, pretrain on simpler synthetic data from tree-grams with moderate n and d to make hyperparameter transfer more robust, then prune up to 95% of routed topologies with no significant performance loss.", "playbook": "training-data", "playbook_url": "https://wpnews.pro/playbooks/training-data"}, "alternates": {"html": "https://wpnews.pro/news/what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of", "markdown": "https://wpnews.pro/news/what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of.md", "text": "https://wpnews.pro/news/what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of.txt", "jsonld": "https://wpnews.pro/news/what-s-in-a-tree-hierarchical-n-gram-models-as-controlled-approximators-of.jsonld"}}