{"slug": "induction-heads-interpolate-n-grams", "title": "Induction Heads Interpolate N-Grams", "summary": "Researchers found that induction heads in transformers implement two smoothing mechanisms—soft context-matching and BOS token pseudo-counts—that interpolate n-gram estimates, enabling in-context learning to regularize rather than simply count. Trained transformers matched or outperformed classical count-based baselines on Markov chain tasks.", "body_md": "arXiv:2607.02800v1 Announce Type: new\nAbstract: Induction heads are attention circuits believed to underlie in-context learning in transformers, yet a precise characterization of the estimators they implement remains elusive. We study transformers trained on order-$k$ Markov chains and identify two complementary smoothing mechanisms. First, at finite attention-weight scale, the circuit implements a soft context-matching estimator: it aggregates contributions from exact and partial context matches, weighted exponentially by their overlap, and induces a data-dependent interpolation across context orders analogous to Jelinek-Mercer smoothing. Second, a beginning-of-sequence (BOS) token induces additive pseudo-counts, recovering Dirichlet-style smoothing. We construct a disentangled transformer implementing both mechanisms and show that trained transformers recover the predicted attention patterns. Across settings where pseudo-count smoothing is optimal or lower-order contexts provide structured evidence, trained transformers match or outperform classical count-based baselines. Our results bridge mechanistic interpretability of induction heads with classical statistical smoothing, revealing that transformers learn to regularize in-context estimation rather than simply count.", "url": "https://wpnews.pro/news/induction-heads-interpolate-n-grams", "canonical_source": "https://arxiv.org/abs/2607.02800", "published_at": "2026-07-07 04:00:00+00:00", "updated_at": "2026-07-07 04:11:37.548895+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "neural-networks", "natural-language-processing", "ai-research"], "entities": ["arXiv", "Jelinek-Mercer", "Dirichlet"], "alternates": {"html": "https://wpnews.pro/news/induction-heads-interpolate-n-grams", "markdown": "https://wpnews.pro/news/induction-heads-interpolate-n-grams.md", "text": "https://wpnews.pro/news/induction-heads-interpolate-n-grams.txt", "jsonld": "https://wpnews.pro/news/induction-heads-interpolate-n-grams.jsonld"}}