Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning

A new inference-time framework, Adjudicated Captioning, achieves CIDEr 117.6 and SPICE 21.9 on the COCO Karpathy test split for zero-shot image captioning, up from 108.0 and 20.3 for the baseline IFCap, a +9.6 CIDEr gain, without retraining the captioner. The method, proposed by researchers on arXiv (2607.28986v1), uses a multi-agent alignment scoring and consensus-distilled beam arbitration, with a training-free fixed-fusion baseline reaching 115.8 CIDEr, indicating that +7.8 of the gain comes from architectural changes and +1.8 from learned rerankers. The recipe also transfers off-COCO, yielding +8.1 CIDEr on Flickr30k Karpathy and +5.7 on NoCaps overall.

arXiv:2607.28986v1 Announce Type: new Abstract: Zero-shot image captioning ZIC describes images without paired image-caption supervision during captioner training, relying on text-only corpora and frozen pretrained image-text scorers. Existing retrieval-augmented methods score image-text alignment once, at retrieval, then commit the captioner's autoregressive beam under language-model probability alone, leaving the decoder without further visual grounding feedback. Progress has stalled, with no method improving on the strict-regime best since 2024. We propose Adjudicated Captioning, an inference-time multi-agent framework that restores grounding feedback at multiple checkpoints over an unchanged IFCap captioner. First, we install a stronger frozen Retrieval Encoder at the input. Second, between retrieval and decoding we insert a frozen Cross-Attention Verifier that re-ranks the top-9 retrievals to top-5. Third, at the output beam we attach a learned Reranker pairing TriFuse, a multilayer perceptron, with MemAttend, a memory-attended transformer, the pipeline's only learned components; both are trained self-supervised by Borda-consensus distillation across the three frozen scorers, using no paired image-caption labels and no reference captions. Under the inductive headline protocol, with rerankers fit on the disjoint COCO Karpathy validation beam and applied frozen to test, the framework reaches CIDEr 117.6 and SPICE 21.9 on COCO Karpathy, up from 108.0 and 20.3 for IFCap, a +9.6 CIDEr gain, and +7.7 above NES, the strongest synthetic-image-augmented method at 109.9, without retraining the captioner. A training-free fixed-fusion baseline reaches 115.8 CIDEr, so +7.8 of the +9.6 gain comes from the non-learned architectural intervention and the remaining +1.8 from the learned rerankers. The same recipe transfers off-COCO without captioner retraining: +8.1 CIDEr on Flickr30k Karpathy and +5.7 on NoCaps overall.