MedTVL: Harnessing Vision and Language for Medical Time Series Classification

Researchers introduced MedTVL, a text-guided dual-pathway architecture for medical time series classification that integrates time series, vision, and language modalities, achieving superior performance across supervised, few-shot, and contrastive learning settings on multiple medical datasets. The model combines a convolution-based temporal pathway and a transformer-based visual pathway, guided by adaptive medical text and a Mixture-of-Experts mechanism to address diagnostic ambiguity and label scarcity.

arXiv:2608.28605v1 Announce Type: new Abstract: Recent advancements in multimodal learning for medical time series MedTS classification highlight the benefits of integrating complementary modalities for clinical decision. However, existing methods typically focus on bi-modal interactions e.g., time series and text , leaving the tri-modal synergy between time series, vision, and language largely unexplored. Inspired by diagnostic practice synergizing numerical assessment, visual inspection and clinical context, we introduce MedTVL, a text-guided dual-pathway architecture tailored for MedTS classification. Specifically, it synergizes a convolution-based temporal pathway for fine-grained temporal dynamics from raw numerical sequences and a transformer-based visual pathway for holistic morphological structures from time-series-derived images. Such combination of cross-modal and architectural heterogeneity provides a comprehensive diagnostic perspective. To further resolve potential diagnostic ambiguity, both pathways are guided by adaptive medical textual semantics. Finally, a Mixture-of-Experts mechanism dynamically routes each instance to specialized fusion experts, capturing instance-specific reliance on the temporal and visual pathway outputs. In addition, MedTVL supports multimodal contrastive learning to mitigate the clinical label scarcity challenge. Extensive experiments across multiple medical datasets and tasks, spanning supervised, few-shot, and contrastive learning settings, demonstrate the superiority and transferability of MedTVL, highlighting its potential for robust clinical decision support.