cd /news/machine-learning/modality-discrepancy-transformer-for… · home topics machine-learning article
[ARTICLE · art-133323] src=arxiv.org ↗ pub= topic=machine-learning verified=true sentiment=↑ positive

Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition

The Modality Discrepancy Transformer (MDT), a conflict-aware multimodal fusion model built on the framework of Bekhouche et al., achieved 0.7408 Macro F1 on the labelled test split and 0.7368 on the private leaderboard of the BAH dataset from the 3rd ABAW Challenge, outperforming the strongest published baseline by more than 10 points. MDT expands the original 6-token design to a 9-token representation combining three modality embeddings, three absolute-difference features, and three Hadamard-product discrepancy features, processed through Transformer self-attention with FiLM-based text-conditioned modulation and LoRA fine-tuning. The model trains in under 20 minutes on a single GPU and uses a text-guided late fusion branch that blends a text-only auxiliary head with the full multimodal output at inference.

by read1 min views2 publishedSep 18, 2026

arXiv:2609.19148v1 Announce Type: new Abstract: Ambivalence and hesitancy (A/H) are affective states in which individuals express contradictory signals across facial, vocal, and linguistic channels. Automatically recognising A/H in clinical videos requires detecting cross-modal disagreement -- the signal that standard fusion methods suppress. Based on the conflict-aware multimodal fusion framework of Bekhouche et al., we present the Modality Discrepancy Transformer (MDT). MDT enriches the original 6-token design to a 9-token representation comprising three modality embeddings, three absolute-difference features, and three Hadamard-product discrepancy features learned through linear projections. These nine tokens undergo Transformer self-attention, with FiLM-based text-conditioned modulation and LoRA fine-tuning as core architectural components. A text-guided late fusion branch blends a text-only auxiliary head with the full multimodal output at inference. On the BAH dataset from the 3rd ABAW Challenge, MDT achieves 0.7408 Macro F1 on the labelled test split and 0.7368 on the private leaderboard, outperforming the strongest published baseline by over 10 points while training in under 20 minutes on a single GPU.

── more in #machine-learning 4 stories · sorted by recency
── more on @modality discrepancy transformer 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/modality-discrepancy…] indexed:0 read:1min 2026-09-18 ·