ReportMedSAM: Guiding Segmentation Through Radiology Reports

Researchers propose ReportMedSAM, a report-driven medical image segmentation framework that uses a learnable concept bank and contrastive learning to align organ-level embeddings with clinical reports, achieving competitive segmentation accuracy on the AbdomenAtlas 3.0 dataset while enabling seamless extension to new tasks without retraining.

arXiv:2607.14116v1 Announce Type: new Abstract: Free-form radiology reports contain rich clinical descriptions, yet converting them for reliable segmentation remains challenging due to the inherent variability of natural language. Existing pipelines often rely on predefined organ phrases or brittle rule-based inference-time extraction, which limits their scalability to novel anatomical structures and makes them sensitive to linguistic variations. To address this, we propose ReportMedSAM, a report-driven framework that replaces discrete extraction with a learnable concept bank. By leveraging a frozen medical vision-language encoder BiomedCLIP , we align organ-level concept embeddings with large-scale clinical corpora through contrastive learning, establishing mutually orthogonal semantic anchors. Our approach explicitly mitigates organ-level semantic collapse and ensures high robustness against diverse clinical synonyms e.g., "renal" vs. "kidney" . During inference, a clinical report is embedded and matched against this concept bank to dynamically activate task-specific Mixture-of-Experts MoE modules. This decoupled design allows new concepts and experts to be added without retraining existing components, providing a parameter-isolated extension mechanism while keeping previously learned experts unchanged. Evaluated on the AbdomenAtlas 3.0 dataset, ReportMedSAM effectively interprets free-form reports, achieves competitive segmentation accuracy, and demonstrates seamless, non-interfering extension to novel clinical tasks.