Estimating the difficulty of medical classification tasks using 3D image datasets

Abstract

Medical deep learning development has seen a rise in recent years. However, for certain applications the model performance remains low compared to other medical applications. To give an early indication of the expected performance for a particular task, developers often start time and resource-consuming benchmark studies, involving training many deep learning models. In this work we instead aim to create a metric for estimating how challenging a classification task is, based on image features of the dataset. Two dataset difficulty metrics, Silhouette score (SIL) and Fréchet inception distance (FID), are applied to 3D medical image classification datasets, based on radiomic features. These metrics are compared to the performance of two deep learning models, to estimate each metric's ability to predict dataset difficulty. The SIL metric shows the strongest correlation to deep learning model performance and indicates potential to estimate dataset difficulty. Further analysis of image feature extraction and difficulty estimation metrics can potentially improve the distinction of datasets in the medical field. This can guide not only the allocation of time and resources on different projects but also further data curation and model development together with clinicians. To promote reproducibility, the code used in this study is available on github at https://github.com/TereseT/Dataset_evaluation.Clinical relevance- This approach can be used to estimate the difficulty of a dataset for computer-aided diagnosis and indicate if further data curation is needed.

Ähnliche Arbeiten