Multimodal Fusion Methods for Whole-Slide Images and Clinical Data in Inflammatory Bowel Disease

A modular framework comparing 15 interchangeable fusion methods for combining whole-slide histology embeddings with per-visit clinical data reached a macro-F1 of 0.57-0.75 in predicting STRIDE-II-aligned remission outcomes for newly diagnosed inflammatory bowel disease patients, according to an arXiv paper (2611.05579v1). The best fusion method outperformed both single-modality models in four of six scenarios, with results varying by outcome and input, and image reliance was highest for histological remission and lowest for endoscopic remission. The authors report that methods with similar accuracy relied on the slide embeddings to very different degrees.

arXiv:2611.05579v1 Announce Type: new Abstract: Multimodal AI fusion of histology and clinical data is an active research area in computational pathology, but fusion methods are rarely compared under identical conditions and across different clinical endpoints. We present a modular framework in which slide-level embeddings, obtained by training an attention-based aggregator over patch embeddings from a frozen foundation model, and per-visit clinical embeddings are fed into 15 interchangeable fusion methods with a shared output dimension and prediction head except decision-level methods . Using a prospective cohort of newly diagnosed inflammatory bowel disease patients, we predicted STRIDE-II-aligned remission outcomes from baseline or follow-up visits. The best fusion method reached macro-F1 of 0.57-0.75, outperforming both single-modality models in four of six scenarios, and differed by outcome and input. Image reliance, the drop in balanced accuracy when the slide embedding is removed, tended to follow the remission type - highest for histological and lowest for endoscopic remission; methods with similar accuracy relied on the slide to very different degrees.