{"slug": "beyond-language-priors-diagnosing-and-fixing-visual-origin-hallucinations-in-llm", "title": "Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM", "summary": "A new study from arXiv (arXiv:2609.00231v1) identifies visual-origin hallucination as a complementary cause of object hallucination in multimodal large language models (MLLMs), showing hallucinated samples have lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns. The researchers propose Adversarial Contrastive Fine-Tuning (ACFT), which uses minimal adversarial perturbations to construct aligned positive-negative pairs, achieving state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL while requiring only 0.9% of the COCO dataset and zero inference overhead.", "body_md": "arXiv:2609.00231v1 Announce Type: new\nAbstract: Existing research on object hallucination in multimodal large language models (MLLMs) predominantly attributes the problem to language priors such as over-reliance on textual co-occurrence statistics. We challenge this view by presenting quantitative evidence for a complementary, under-explored cause: visual-origin hallucination, where hallucinations arise from incorrect visual feature extraction and misalignment between image and text embeddings. Through cosine similarity analysis and Smooth Grad-CAM entropy measurements, we show that hallucinated samples exhibit systematically lower image-text similarity (average 0.158 vs. -0.122) and inverted attention patterns, where attention is dispersed when the target object is present but wrongly concentrated when it is absent. Guided by this diagnosis, we propose Adversarial Contrastive Fine-Tuning (ACFT). ACFT uses an Adversarial Hallucination Attribute Flipping (AHAF) procedure, involving minimal, targeted adversarial perturbations that flip an image's hallucination attribute, to construct perfectly aligned positive-negative pairs, which are then used for contrastive fine-tuning. AHAF simultaneously serves as a diagnostic probe, revealing that MLLM visual representations lie dangerously close to hallucination decision boundaries. Requiring only 0.9% of the COCO dataset and adding zero inference overhead, ACFT achieves state-of-the-art performance on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL. Code is available at https://github.com/zxp555/ACFT_MM", "url": "https://wpnews.pro/news/beyond-language-priors-diagnosing-and-fixing-visual-origin-hallucinations-in-llm", "canonical_source": "https://arxiv.org/abs/2609.00231", "published_at": "2026-09-02 04:00:00+00:00", "updated_at": "2026-09-02 04:22:57.979900+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-safety"], "entities": ["arXiv", "ACFT", "COCO", "POPE", "MME", "LLaVA", "MiniGPT-4", "Qwen2.5-VL"], "alternates": {"html": "https://wpnews.pro/news/beyond-language-priors-diagnosing-and-fixing-visual-origin-hallucinations-in-llm", "markdown": "https://wpnews.pro/news/beyond-language-priors-diagnosing-and-fixing-visual-origin-hallucinations-in-llm.md", "text": "https://wpnews.pro/news/beyond-language-priors-diagnosing-and-fixing-visual-origin-hallucinations-in-llm.txt", "jsonld": "https://wpnews.pro/news/beyond-language-priors-diagnosing-and-fixing-visual-origin-hallucinations-in-llm.jsonld"}}