AdaptSAM: Adaptive SAM for Cross-Domain Few-Shot Medical Image Segmentation

Abstract

Deep networks excel in medical image segmentation with large annotated datasets but struggle with generalization to unseen out-of-domain data, a common challenge in clinical settings. Although the Segment Anything Model (SAM) demonstrates strong prompt-driven generalization in natural image segmentation, its application to clinical segmentation, especially in cross-domain and few-shot scenarios, faces limitations due to insufficient domain-specific features, weak structural fidelity, and prompt instability. To address these challenges, we propose AdaptSAM, an innovative framework to improve cross-domain few-shot medical image segmentation through three key components: Frequency-aware, Semantically-aligned, and Promptadaptive strategies. The Frequency-domain Multi-scale Feature Enhancement (FFE) module extracts frequency-aware features and enriches them with context-sensitive semantics, mitigating domain-specific feature loss. The enhanced features are fused with the Hierarchical Semantic Refinement (HSR) module, utilizing high-level semantic activations to recalibrate shallow-layer features, improving fine-structure fidelity and boundary preservation. Additionally, the Dynamic Curriculum Prompt (DCP) mechanism adjusts prompt box sizes during training, guiding the model to learn object-boundary interactions and background context in a coarse-to-fine manner, thereby aligning prompts with target domain features and enhancing segmentation robustness across domain shifts. AdaptSAM outperforms state-of-the-art methods on ten public medical segmentation datasets, achieving 6.22 % higher Dice than SAM-based methods and 10.37 % higher than fully supervised domain adaptation, showcasing superior cross-domain generalization with just one labeled sample. Code is provided at: https://github.com/Ggllllllll/AdaptSAM.

Ähnliche Arbeiten