DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

Researchers introduced Dual-Entropy Enhanced Policy Optimization (DEEPO), a dual-stage reinforcement learning method that reduces hallucination in multimodal large language models by combining semantic-entropy-triggered expert prefixes with advantage-sign-aware Renyi gradient preconditioning. DEEPO's two branches each improved over GRPO individually, and their interaction was statistically significant on VideoMMMU, the most complex long-horizon task in the evaluation suite, at +4.0 points (95% CI [1.1, 6.9]), with additive gains elsewhere. The method reduces hallucination while preserving accuracy and training stability, according to the arXiv:2609.28570v1 paper.

arXiv:2609.28570v1 Announce Type: new Abstract: Reinforcement learning RL is widely used to sharpen reasoning in multimodal large language models MLLMs , yet its effect on hallucination is uneven. We trace this to two weak points in the \emph{correction chain} from reward to parameter update. At the rollout level, hard queries---those with high semantic entropy---frequently produce unanimously wrong sample groups, collapsing the group-relative advantage to zero exactly where hallucination risk is highest. At the optimization level, confident-but-wrong tokens are gradient-invisible: a categorical policy's expected score-gradient norm vanishes as its distribution sharpens, so the predictions that most need correction receive the weakest updates. We propose Dual-Entropy Enhanced Policy Optimization DEEPO , a dual-stage enhancement combining signal variance regularization with gradient preconditioning: semantic-entropy-triggered expert prefixes inject grounded continuations on high-uncertainty queries, providing direct supervision and restoring advantage variance, while advantage-sign-aware Renyi preconditioning counteracts logit-level saturation so correction reaches confident errors in the operational confidence regime. Both branches improve over GRPO individually; their interaction is statistically significant on VideoMMMU---the most complex long-horizon task in our evaluation suite +4.0$, 95\% CI 1.1, 6.9 ---and additive elsewhere. DEEPO reduces hallucination while preserving accuracy and training stability.