Surgical Action Collaboration Through Multimodal Large Language Model-Driven Surgical Tool Dialogue in Robotic Assisted Surgery

Abstract

Recent advances in multimodal large language models (MLLMs) have demonstrated significant potential for complex reasoning and decision-making tasks. In robotic surgery, intuitive instrument coordination is critical, yet existing systems primarily rely on high-level individual tool commands without addressing real-time multi-tool collaboration. Current surgical datasets lack annotations for inter-tool interactions, limiting the development of collaborative surgical frameworks. Here we present a multi-agent MLLM-based surgical tools dialogue architecture that enables decentralized action planning and centralized coordination through specialized tool agents and a master arbitrator. We enhanced the SAR-RARP50 dataset with fine-grained collaborative action labels, creating the SAR-RARP50-SAC dataset with 20 action categories across 8 surgical tools. Our multi-agent system achieved 55.44% overall accuracy with Chain-of-Thought prompting and 48.08% without CoT, significantly outperforming the single-agent baseline (33.62%). Individual tool performance varied from 19.62% to 99.77%, with specialized tools like the clip applier showing superior accuracy. These results demonstrate that multi-agent architectures effectively leverage MLLM reasoning capabilities for surgical tool coordination, enabling explicit inter-tool communication and collaborative decision-making in robotic-assisted surgery. The source code and dataset will be available.

Ähnliche Arbeiten