MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

A new study from arXiv (2609.04336v1) introduces MedProb, a lightweight probing framework that predicts multiple-choice medical visual question answering (Med-VQA) answers from frozen vision-language model (VLM) representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and outperforms medical VLMs and agentic systems, while also reducing the apparent performance gap between small and large models. The study also finds that medical adaptation does not consistently improve linear decodability across 14 matched VLM pairs and that free-text generation exhibits an answer-position bias of up to 10 percentage points.

arXiv:2609.04336v1 Announce Type: new Abstract: Medical visual question answering Med-VQA is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with \textbf{MedProb}, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals. Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability. Finally, free-text generation exhibits an answer-position bias of up to 10 percentage points, whereas MedProb also has positional bias, however, it is impacted differently than prompting. Our main results target the multiple-choice/multiclass Med-VQA setting; we additionally show the probe can be extended to open-ended generation via a rejection-sampling scoring procedure.