MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

Researchers introduced MMLongEmbed, the first comprehensive benchmark for evaluating multimodal embedding models in long-context scenarios. The benchmark includes four retrieval tasks across text, document, and video modalities, revealing that current models rely on superficial feature matching and struggle with deep semantic dependencies. Performance degradation varies with context length and key information placement, and models show different robustness to redundancy across modalities.

arXiv:2606.14747v1 Announce Type: new Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models MEMs . However, larger context windows do not necessarily translate into effective comprehension and representation of long-context multimodal inputs, which remains a critical bottleneck for real-world deployment. To address the lack of systematic evaluation in this setting, we introduce MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios. MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities. Through extensive evaluation of state-of-the-art models, we find that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies. We further observe that performance degradation varies systematically with context length and key information placement. Moreover, models exhibit substantially different robustness to redundant contextual information across modalities. For reproducibility, the benchmark and code are publicly available.