{"slug": "which-modality-decides-counterfactual-modality-attribution-for-multimodal-llms", "title": "Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs", "summary": "Researchers introduced Counterfactual Modality Attribution (CMA), the first framework for quantifying modality-level contributions in multimodal large language models (MLLMs), which identifies the decision-driving modality in 98% of controlled cases. The method, detailed in a paper on arXiv (2608.00076v1), uses coupled diffusion priors and Shapley values to generate counterfactuals and attribute predictions to image or text, revealing cross-modal reasoning failures invisible to accuracy metrics. The authors position CMA as a complementary explainability tool for auditing multimodal foundation models in safety-critical applications.", "body_md": "arXiv:2608.00076v1 Announce Type: new\nAbstract: Multimodal large language models (MLLMs) increasingly support high-stakes decision making by combining complementary information from images and text. While existing explainability methods identify influential image regions or text tokens, they cannot answer a fundamental question: which modality drives a prediction? Consequently, a model may produce the correct output while relying on the wrong source of evidence, masking shortcut learning and unsafe reasoning. We formulate modality attribution as a complementary explainability objective for multimodal foundation models and propose Counterfactual Modality Attribution (CMA), the first framework for quantifying modality-level contributions in MLLMs. CMA generates image-only, text-only, and joint multimodal counterfactuals using coupled diffusion priors and converts them into principled modality attribution scores through a cooperative game-theoretic formulation based on Shapley values. We evaluate CMA on controlled synthetic benchmarks with known ground-truth modality reliance and on a real-world multimodal clinical dataset. CMA correctly identifies the decision-driving modality in 98% of controlled cases and consistently outperforms baselines, revealing failures of cross-modal reasoning that remain invisible to predictive accuracy alone. Our results establish modality attribution as a complementary dimension of explainability beyond feature attribution, providing a principled framework for auditing multimodal foundation models in safety-critical applications.", "url": "https://wpnews.pro/news/which-modality-decides-counterfactual-modality-attribution-for-multimodal-llms", "canonical_source": "https://arxiv.org/abs/2608.00076", "published_at": "2026-08-04 04:00:00+00:00", "updated_at": "2026-08-04 04:38:39.105076+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-safety"], "entities": ["Counterfactual Modality Attribution", "arXiv", "multimodal large language models"], "alternates": {"html": "https://wpnews.pro/news/which-modality-decides-counterfactual-modality-attribution-for-multimodal-llms", "markdown": "https://wpnews.pro/news/which-modality-decides-counterfactual-modality-attribution-for-multimodal-llms.md", "text": "https://wpnews.pro/news/which-modality-decides-counterfactual-modality-attribution-for-multimodal-llms.txt", "jsonld": "https://wpnews.pro/news/which-modality-decides-counterfactual-modality-attribution-for-multimodal-llms.jsonld"}}