MP41-03 COMPUTER VISION FOR SURGICAL ERRORS DETECTION DURING ROBOTIC TISSUE DISSECTION

Abstract

You have accessJournal of UrologyCME1 May 2022MP41-03 COMPUTER VISION FOR SURGICAL ERRORS DETECTION DURING ROBOTIC TISSUE DISSECTION Rafal Kocielnik, Inhoo Lee, Jasper Laca, Sidney I. Roberts, Jessica H. Nguyen, Idris O. Sunmola, De-An Huang, Sandra P. Marshall, Anima Anandkumar, and Andrew J. Hung Rafal KocielnikRafal Kocielnik More articles by this author , Inhoo LeeInhoo Lee More articles by this author , Jasper LacaJasper Laca More articles by this author , Sidney I. RobertsSidney I. Roberts More articles by this author , Jessica H. NguyenJessica H. Nguyen More articles by this author , Idris O. SunmolaIdris O. Sunmola More articles by this author , De-An HuangDe-An Huang More articles by this author , Sandra P. MarshallSandra P. Marshall More articles by this author , Anima AnandkumarAnima Anandkumar More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000002607.03AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Real-time detection of errors during robot-assisted surgery (RAS) currently requires supervision from experienced surgeons. Herein, we attempt to automate surgical error detection through machine learning (ML)-based computer vision. METHODS: We use RAS data from 23 surgeons performing a simulated dry-lab tissue dissection task on a live daVinci Xi surgical robot. The data contains video and per-second Index of Cognitive Activity (ICA), derived from pupillary change and tracked by Tobii Eyetrackers. In this data, 172 error instances, representing 11 error types, have been previously annotated. Of these, Tissue tears (n=70, 40.7%) and Tissue punctures (n=62; 36%) are the 2 most common error types. We extract 5 sec windows around an error (positive label) and randomly sample the remaining data (negative label). We train 3 variations of ML architectures on a 80/20 train-test random split stratified by label and participant. Our base 1-stream Long Short-term Memory (LSTM) ML model relies on RGB (red green blue) video input only (Figure 1.1a). Our 2-stream model expands on the base model to include movement information via Optical Flow (Figure 1.1b) and a Spatial Attention Mechanism (Figure 1.1c). Finally, the 3-stream model additionally includes surgeon ICA data (Figure 1.1d). RESULTS: For detection of “Any error”, additional data (movement+ICA) improve performance from AUC 0.60 (1-stream) to 0.70 (3-stream; Figure 1.2). Between the two most common error types, detection of “Tissue tears” benefits from ICA data, which may indicate the importance of the surgeon's state of mind while committing this type of error. In contrast, detection of “Tissue punctures” degrades from 0.71 to 0.57 with addition of attention mechanism and movement data. This suggests conflicting information added or due to these errors being less visually apparent, which may make it challenging for the ML attention mechanism to identify them. CONCLUSIONS: We have demonstrated the feasibility of ML-based automated error detection in RAS. Our exploration demonstrates the positive value of including additional information in certain tasks. Future work will explore better mechanisms of combining multiple information sources and also expand the setup to detection and possibly prediction of a broader set of errors. Source of Funding: None © 2022 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 207Issue Supplement 5May 2022Page: e719 Advertisement Copyright & Permissions© 2022 by American Urological Association Education and Research, Inc.MetricsAuthor Information Rafal Kocielnik More articles by this author Inhoo Lee More articles by this author Jasper Laca More articles by this author Sidney I. Roberts More articles by this author Jessica H. Nguyen More articles by this author Idris O. Sunmola More articles by this author De-An Huang More articles by this author Sandra P. Marshall More articles by this author Anima Anandkumar More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF DownloadLoading ...

Ähnliche Arbeiten