arXiv:2611.17585v1 Announce Type: new Abstract: Reward models are a core component of modern text-to-image systems, serving as the objective for inference-time selection and RL-based post-training. A key prerequisite is visual understanding: to judge prompt adherence and ground aesthetic preferences, a reward model must bind attributes, resolve relations, and count reliably. We find that widely-used reward models perform poorly on targeted visual reasoning evaluations. Even when starting from strong vision-language backbones, preference fine-tuning erodes these capabilities, and continual-learning baselines fail to mitigate a steep tradeoff between visual reasoning and preference accuracy. We propose a data-centric intervention that injects visual reasoning supervision into preference learning by flipping the pairing axis. To complement conventional human preference data (paired images under a shared prompt), we convert existing visual reasoning datasets into preference pairs that share the same image but differ in prompt, and train on the mixture. This yields a Pareto improvement in both visual understanding and preference prediction. Across inference-time best-of-n selection and RL post-training, the resulting reward model improves text-to-image generation on challenging compositional prompts (GenEval) by an average of 4.0% and 5.5% respectively, outperforming HPSv3 which was trained on 6x more preference pairs.
Rethinking Visual Reasoning in Text-to-Image Reward Modeling
A data-centric training intervention that converts existing visual reasoning datasets into preference pairs sharing the same image but differing in prompt produces a Pareto improvement in both visual understanding and preference prediction for text-to-image reward models, according to arXiv paper 2611.17585v1. The resulting reward model improves text-to-image generation on challenging compositional prompts in GenEval by an average of 4.0% for inference-time best-of-n selection and 5.5% for RL post-training, outperforming HPSv3, which was trained on 6x more preference pairs. The authors found that widely-used reward models perform poorly on targeted visual reasoning evaluations, and that preference fine-tuning erodes visual reasoning even from strong vision-language backbones, with continual-learning baselines failing to mitigate the steep tradeoff between visual reasoning and preference accuracy.
Run your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.