Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation

Researchers propose Dual-Adaptive SAM3 (DA-SAM3), a framework for parameter-efficient medical image segmentation that uses a Dynamic Expert Router and Decomposed Parameterized Experts to reduce MoE overhead by over 80% while achieving state-of-the-art accuracy. The method matches or exceeds fully fine-tuned SAM3 and standard MoE baselines, with a 5% gain over current methods on public benchmarks.

arXiv:2607.02571v1 Announce Type: new Abstract: The Segment Anything Model with Concepts SAM3 heralds a new paradigm for open-vocabulary segmentation through natural language interaction, offering significant potential for medical image analysis. However, effectively adapting such a powerful vision-language model to the diverse and nuanced domain of medical imaging remains a key challenge. Naive fine-tuning is parameter-inefficient, while standard Mixture-of-Experts MoE methods introduce prohibitive computational overhead, limiting their clinical applicability. To address this, we propose Dual-Adaptive SAM3 DA-SAM3 , a novel framework that achieves both high segmentation accuracy and extreme parameter efficiency via a dual-adaptive specialization mechanism. Our first adaptation is task-aware: a Dynamic Expert Router DER that sparsely activates the most relevant experts by jointly reasoning about the visual input and the textual concept prompt, mimicking a clinical consultation process. Our second adaptation is parameter-aware: a Decomposed Parameterized Experts DPE design that represents each expert as a shared frozen base inherited from the pretrained SAM3 and a lightweight trainable low-rank delta, reducing MoE parameter overhead by over 80\%. Extensive experiments on multiple public medical segmentation benchmarks demonstrate that Dual-Adaptive SAM3 not only matches or exceeds the accuracy of fully fine-tuned SAM3 and standard MoE baselines, but also achieves a notable 5\% gain over current state-of-the-art methods, with interpretable results validating its effectiveness. The code is available at: https://github.com/Reconsider80/DA-SAM3.