RSG-SAM2: Robust Semantics-Guided SAM2 for Multi-Class Surgical Instruments Semantic Segmentation

Abstract

Recent advances in segmentation foundation models, especially Segment Anything Model 2 (SAM2) and its derivatives such as MedSAM2 and Surgical-SAM2, have demonstrated remarkable performance across various medical image segmentation tasks. However, these models still exhibit limited semantic robustness, relying primarily on low-level prompts or structural priors rather than semantic understanding. Such limitations reduce their effectiveness in multi-class surgical instrument segmentation and restrict their applicability to autonomous endoscopic scene analysis. To address these issues, we introduce RSG-SAM2 (Robust Semantics-Guided SAM2), a novel framework designed to strengthen both the semantic perception and domain adaptability of SAM2 within surgical semantic segmentation. RSG-SAM2 establishes a dual-stage semantic guidance mechanism that integrates early-stage text-driven feature fusion with late-stage semantic alignment between visual and textual representations. This design enables the model to robustly capture fine-grained semantic distinctions among similar surgical instruments, improving category discrimination and stability under complex surgical conditions. With full encoder fine-tuning and semantic guidance, RSG-SAM2 achieves a Challenge IoU of $\mathbf{8 3. 8 4 \%}$, surpassing the state-of-the-art QPD method on the EndoVis-2018 benchmark.

Ähnliche Arbeiten