A MULTIMODAL NEURAL NETWORK IMPROVES PREDICTIONS OF OVERALL SURVIVAL AND RISK OF METASTASES IN PATIENTS WITH SOFT-TISSUE SARCOMA

Abstract

The 5-year survival of patients with soft tissue sarcoma (STS) is only 65% and has not improved over the past 3 decades. New innovations in matching patients to optimal treatments are needed to improve outcomes. While not indicated for all STS patients, the decision to use chemotherapy is influenced by the predicted aggressivity of a patient's sarcoma, and thus the patient's risk of developing metastases. Furthermore, the ideal surveillance regimen may be one tailored to a patient's perspectives and individualized risk of metastases. Whereas currently available nomograms use only clinical variables and cannot guide individualized management, the inclusion of rich data like MRIs in prediction models may result in predictions accurate enough to enable individualized treatment. The objective of this study is to develop a multimodal neural network model (MMNN) that analyzes clinical variables as well as MRI images of a sarcoma, to predict an STS patient's overall survival and risk of distant metastases. We compare the performance of this MMNN to other models based on clinical variables alone, radiomics models, and a unimodal neural network. All patients aged 18 or older with biopsy-confirmed non-retroperitoneal STS who underwent primary resection at MSKCC between January 1st, 2005, and December 31st, 2020 were reviewed. We included all patients with complete outcome data and a pre-treatment MRI with both a T1 post-contrast sequence and a T2 fat-sat sequence available. Preprocessing of the MRI data included N4 bias correction and z-score normalization. A total of 9380 MRI slices containing sarcomas are available for analysis. Our MMNN [Figure 1] accepts the T1 and T2 MRIs which are fed through an image subnetwork consisting of a 2-channel DenseNet-121. The T1 and T2 sequences are masked and cropped to contain only the tumor volume. Clinical variables are analyzed in a parallel deep neural network and this information is concatenated with the image features. A fully connected layer analyzes the combined multimodal features before outputting the predicted risk of each of our outcomes. Gradient blending is used to moderate the loss contributions of the different modalities during the training of multimodal neural networks. Visualization of the image features using heat maps was obtained using the Grad-CAM methodology. Our MMNN outperformed all other models in predicting overall survival and the risk of distant metastases [Table 1]. The C-Index of our MMNN for overall survival is 0.769, an absolute increase of 11.4% in AUC compared to the next best performing model. The heat maps demonstrate the areas of the sarcomas deemed most salient for the predictions [Figure 2]. This is the first multimodal neural network in sarcoma. Given the rarity of STS, the use of multimodal data in prediction algorithms is essential to overcoming limits of small sample sizes. Future work will seek to externally validate this model using federated learning. For any figures or tables, please contact the authors directly.

Ähnliche Arbeiten