TRACE: Training-aware Robustness and Challenge Evaluation for Text-Vision Embedding Models

A new arXiv paper (2610.26993v1) introduces TRACE, a Training-aware Robustness and Challenge Evaluation framework that reorganizes 168 tasks into General, Adaptability, and Challenge categories to separate general capability from task-specific adaptation and robustness in text-vision embedding models. In controlled experiments on two text-vision embedding models, combining general and adaptation data improved Adaptability performance by 7.67 and 7.09 points while largely preserving General performance, but Challenge performance stayed below 5.0 with limited gains from additional training data. The authors identify multilingual and visual-document tasks as major weaknesses and conclude that data adaptation alone is insufficient, calling for methodological advances beyond data-centric improvements.

arXiv:2610.26993v1 Announce Type: new Abstract: Existing benchmarks for text-vision embedding models report aggregate scores that obscure whether performance comes from general capabilities, task-specific adaptation, or robustness to challenging tasks. We introduce TRACE, a Training-aware Robustness and Challenge Evaluation framework that reorganizes 168 tasks from various sources of datasets into three categories: General, Adaptability, and Challenge. Through controlled experiments on two text-vision embedding models, we show that combining general and adaptation data improves Adaptability performance by 7.67 and 7.09 points, respectively, while largely preserving General performance. However, Challenge performance remains below 5.0, with limited improvements from additional training data. Further analysis identifies multilingual and visual-document tasks as major weaknesses. Our findings demonstrate that data adaptation alone is insufficient to address these challenges, highlighting the need for methodological advances beyond data-centric improvements.