{"slug": "causalgate-causal-importance-distillation-for-transformer-module-pruning", "title": "CausalGate: Causal Importance Distillation for Transformer Module Pruning", "summary": "Researchers introduced CausalGate, an intervention-guided framework for transformer module pruning that measures semantic damage via Kullback-Leibler divergence, outperforming dynamic routing baselines on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B with zero operational overhead.", "body_md": "arXiv:2607.22720v1 Announce Type: new\nAbstract: Existing adaptive inference methods for Large Language Models rely on observational heuristics, such as hidden-state similarity or activation magnitudes, to drop redundant modules. However, these correlation-based metrics often fail to capture subtle, non-linear structural computations vital for semantic accuracy. We introduce CausalGate, an intervention-guided framework for compute-efficient transformer inference. During a calibration phase, CausalGate isolates individual Attention and MLP sub-layers, zeros out their respective outputs, and measures the exact semantic damage via the Kullback-Leibler divergence of the final logit distribution. To eliminate runtime routing overhead, this structural importance hierarchy is distilled into a global set of static, lightweight scalar gates using an Exponential Moving Average smoothing objective paired with a differentiable pairwise ranking loss. Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B across language modeling and commonsense reasoning benchmarks, CausalGate consistently outperforms prominent dynamic routing and layer-skipping baselines, translating theoretical compute savings into concrete hardware latency reductions with zero operational overhead.", "url": "https://wpnews.pro/news/causalgate-causal-importance-distillation-for-transformer-module-pruning", "canonical_source": "https://arxiv.org/abs/2607.22720", "published_at": "2026-07-28 04:00:00+00:00", "updated_at": "2026-07-28 04:12:09.227190+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "machine-learning", "ai-research"], "entities": ["CausalGate", "TinyLlama-1.1B", "Qwen2.5-3B", "Llama-3.1-8B"], "alternates": {"html": "https://wpnews.pro/news/causalgate-causal-importance-distillation-for-transformer-module-pruning", "markdown": "https://wpnews.pro/news/causalgate-causal-importance-distillation-for-transformer-module-pruning.md", "text": "https://wpnews.pro/news/causalgate-causal-importance-distillation-for-transformer-module-pruning.txt", "jsonld": "https://wpnews.pro/news/causalgate-causal-importance-distillation-for-transformer-module-pruning.jsonld"}}