Inference-Time Agentic Decision Rules Beat Longer Evolving Search for Multi-Image Medical Reasoning

A new arXiv study (2607.27564v1) finds that a simple order-vote policy outperforms more complex agentic strategies for multi-image medical visual question answering, achieving 57.89% ± 0.65% final-test accuracy on MedFrameQA, compared to 52.73% ± 0.42% for the fixed baseline and 55.79% ± 0.43% for the order-rerank variant. Extending the evolutionary search budget from 50 to 100 generations did not improve generalization, with final-test accuracy dropping to 56.02%, indicating that the choice of agentic decision rule matters more than search budget.

arXiv:2607.27564v1 Announce Type: new Abstract: Multi-image medical VQA is not merely a prompt-length problem; it is a fundamental challenge of agentic decision-making. Medical vision-language agents must aggregate evidence across ordered images, remain robust to answer-order perturbations, and avoid overfitting to noisy search-time feedback. We study MedFrameQA through a controlled comparison of five inference-time agentic strategies, optimized using the same high-budget ShinkaEvolve configuration and evaluated on a reproducible internal frozen split 1,331 evolution, 665 holdout, 855 final test . Across five independent repeated runs, the strongest method emerges as the simplest robust aggregator: the \textbf{order-vote} policy achieves $57.89 \pm 0.65\%$ final-test accuracy, significantly outperforming the fixed baseline $52.73 \pm 0.42\%$ and the more complex, albeit brittle, order-rerank variant $55.79 \pm 0.43\%$ . Paired bootstrap analysis confirms these significant gains. Extending the evolutionary search budget from 50 to 100 generations yields no generalization benefit: while holdout performance marginally increases, final-test accuracy drops from $57.89\%$ to $56.02\%$. Our findings suggest that for multi-image medical reasoning, defining the correct agentic decision rule is substantially more impactful than expanding the optimization search budget.