Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs

Researchers have identified three critical limitations in existing preference optimization methods for medical Large Vision-Language Models (LVLMs), including their inability to distinguish clinically critical tokens from filler text and a lack of visual grounding constraints. The team proposes a new alignment framework using a bidirectional token-wise KL regularizer and a visual-contrastive grounding objective that penalizes responses lacking adequate visual evidence. This approach improves clinical accuracy by correcting only erroneous spans in model outputs while preserving linguistic style, as validated across multiple medical imaging tasks.

arXiv:2606.12590v1 Announce Type: new Abstract: Large Vision-Language Models LVLMs have achieved strong performance across medical imaging tasks, yet they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback. Existing post-training alignment approaches, including Direct Preference Optimization DPO and its variants, face three critical limitations in the medical domain: 1 sequence-level reward signals treat clinically critical tokens identically to generic filler text; 2 reliance on static supervised fine-tuning references as preferred responses introduces an off-policy distribution shift, steering optimization toward stylistic artifacts over clinical correctness; and 3 alignment objectives lack explicit visual grounding constraints, leaving models insensitive to subtle yet diagnostically decisive pathological features. Our method leverages a bidirectional token-wise KL regularizer alongside a visual-contrastive grounding objective that pairs clean and lesion-corrupted images to penalize responses generated without adequate visual evidence. Together, these components form a fine-grained, on-policy alignment framework that constructs preference pairs by minimally editing model-generated outputs, correcting only clinically erroneous spans while preserving the original linguistic style. Extensive experiments across medical imaging tasks and clinical text generation benchmarks validate the effectiveness of our approach.