cd /news/large-language-models/what-s-in-a-tree-hierarchical-n-gram… · home › topics › large-language-models › article
[ARTICLE · art-172362] src=machinebrief.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

Researchers introduced tree-grams, autoregressive generative n-gram models that build the conditional probability tensor p(x_n | x_1:n-1) through hierarchical composition of order-2 and order-3 tensors, in arXiv paper 2611.23093v1. Tree-grams outperform existing n-gram baselines while remaining competitive with deep transformer-based language models, and the authors report that routing across candidate topologies produces emergent sparsity allowing pruning of up to 95% of topologies without significant performance change. The work also shows hyperparameter transfer under μP becomes more robust when narrower models are tuned on simpler data generated by tree-grams with moderate n and d, with entropy and effective correlation range controlling the performance gap after fine-tuning.

recommendation

When tuning narrower models under μP, pretrain on simpler synthetic data from tree-grams with moderate n and d to make hyperparameter transfer more robust, then prune up to 95% of routed topologies with no significant performance loss.

more in playbook: Fine-tuning and data
by read1 min views3 publishedNov 19, 2026

arXiv:2611.23093v1 Announce Type: new Abstract: Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $\mu$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95% of topologies without significant change in performance.

── more in #large-language-models 4 stories · sorted by recency
── more on @tree-grams 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/what-s-in-a-tree-hie…] indexed:0 read:1min 2026-11-19 · —