Authors’ reply to Kulkarni

Abstract

We thank Kulkarni[1] for his comments on our article titled, “Novel artificial intelligence (AI) algorithm for automatic detection of COVID-19 abnormalities in computed tomography images.”[2] We will attempt to respond to the queries raised in the letter. Regarding the question whether a pre-trained model was used, and the request to provide additional details, we have used a pre-trained RetinaNet model which used imagenet data for training and in addition, performed transfer learning for medical images. Due to the unavailability of pre-trained models on COVID-19 image data at the time of our study, we did not use specifically pre-trained models. In our future studies, we will perform the analysis according to insightful suggestions by the reviewer. With regard to the question about the baseline model, suggestions can be used for comparison for different CNN models. The comparison will also show the effectiveness of the RetinaNet model as compared to other architectures like ResNets, DenseNet, VGG, etc. However, we have used RetinaNet, because it had single-shot detection capabilities and effectiveness on object detection as compared to other methods. Since the objective of our study was to estimate the effectiveness and baseline performance of deep learning in the detection of COVID-19 infection and to play the role of assistive technology,[34] we have not conducted such comparisons. We plan to include such comparisons in our future studies. Regarding the question about the confusion matrix, we agree with Kulkarni on the need for extensive quantitative evaluation of the results to help medical practitioners in making an informed decision. At the time of our study, we had limited data and extensive analysis was not possible. Since the objective was to measure the quantitative and qualitative effectiveness of deep learning-based AI in a clinical study, we had compared the images with radiologist labels and added a qualitative analysis of the results. Detailed quantitative analysis based on demography, medical history and AI mistakes will be useful for new insights on AI capabilities. However, this would require additional annotations. With the availability of more data and corresponding annotations, we plan to conduct such a study in the future. Regarding our decision to use RetinaNet rather than other models, at the time of our study (late 2020), our objective was to provide the effectiveness of the method for the purpose of COVID-19 detection and to evaluate the use of AI in the setting of a clinical study. The suggestion from Kulkarni highlights a very crucial point and it can be used to measure the performance of RetinaNet against other models. Since the performance of the model in training and validation was satisfactory, we have treated this model as the baseline and conducted the study. However, we agree with Kulkarni, and in future, we plan to compare the performance of different models to further improve the performance. Regarding the comment that there may have been bias in the dataset on account of the fact that we collected the normal (non-COVID) images from those of patients of a tertiary cancer care center, we have mentioned process of the COVID-19 negative data collection in the data creation section of the manuscript. We collected all the data of normal and non-COVID-19 patients (other abnormalities) during the period of the study. The collected data were used without any filter on these patients; hence, it demonstrates close to the real-world performance at the Tata Memorial Hospital, Mumbai, India. Regarding the biases due to the demographic profiles of the patients, we had already discussed this in the paper, and we plan to conduct more extensive studies to quantitatively estimate the bias in the AI. Regarding the bounding box performance measures, we agree with Kulkarni on the qualitative measure of the bounding box. However, we have used the bounding box predictions for qualitative evaluation of the AI. We have added the results of the same to the discussion in our paper. Since this requires an additional annotation effort, we plan to conduct such extensive studies in future to estimate the performance of the bounding box predictions. Finally, we agree with Kulkarni that cross-validation will provide a more detailed and robust performance of the model. However, in this study, we used fixed training and validation data for evaluation. The results were further qualitatively verified by the radiologist to measure the effectiveness.[56] For future studies, we will include the cross-validation as suggested. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.

Ähnliche Arbeiten