Comprehensive Quantitative Evaluation of the Performance and Trustworthiness of Deep Learning Models - Skin Lesion Classification Case Study

Abstract

Skin cancer remains a widespread and potentially deadly disease, where timely and accurate diagnosis significantly improves patient outcomes. However, the diagnostic process encounters challenges due to substantial variability within lesion classes and similarities among different lesion types. Publicly available datasets, such as dermoscopic image collections, commonly exhibit imbalanced distributions, notably underrepresenting malignant lesions, further complicating the development and validation of accurate classification systems. Additionally, there is a scarcity of comprehensive and standardized evaluation metrics reported in existing research. This study addresses these limitations by comprehensive evaluation of a multi-class deep neural network classifier trained on dermoscopic images, using rigorous cross-validation and independent external testing to mitigate dataset imbalance and ensure reliable performance assessment. Furthermore, to make the model more understandable and thus more useful, the number of explainability techniques has been compared. Apart from visual comparison of the various XAI methods, their quantitative evaluation using 5 metric categories: Faithfulness, Robustness, Localisation, Complexity, and Randomisation, was performed. The evaluation of the proposed deep neural network model for classification revealed its noteworthy performance, with accuracy, F1-score, precision, and recall values of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$77.60 \pm 0.67,70.25 \pm 1.09,72.69 \pm 1.28$</tex>, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$70.20 \pm 1.18$</tex>, respectively. This places the model competitively alongside contemporary state-of-the-art methods.

Ähnliche Arbeiten