How to Determine If One Diagnostic Method, Such as an Artificial Intelligence Model, is Superior to Another: Beyond Performance Metrics

Abstract

Take-home points The effects of a diagnostic method, such as an artificial intelligence (AI) model, on patient outcomes cannot be determined by analyzing performance metrics (such as the area under the receiver operating characteristic curve, sensitivity, specificity, or the Youden index) alone. Two diagnostic methods can be compared more holistically in relation to patient outcomes using an equation, sensitivity x prevalence + specificity x (1 -prevalence) x false positive (FP)to-true positive (TP) outcome ratio, derived using the definition of the net benefit in the decision curve analysis, where the "FP-to-TP outcome ratio" is the ratio between the absolute amounts of net loss in patient outcomes incurred by an FP decision instead of leaving the patient alone and the net outcome gain provided by a TP decision compared with neglecting the disease in the patient. The equation can be useful for a preliminary estimation of the effects of a diagnostic method, such as AI, on patient outcomes when direct data are not available.

Ähnliche Arbeiten