Latent Supervision: A Method for Improved Performance and Calibration of Machine Learning Classification Models in Ophthalmology

Abstract

Purpose: Standard supervised learning assumes deterministic labels (e.g., positive or negative, present or absent), neglecting the diagnostic uncertainty inherent in clinical practice.We propose latent supervision, a novel algorithm which applies latent class analysis (LCA) to incorporate multiple diagnostic tests or expert opinions to produce probabilistic labels (soft labels) for more accurate, calibrated ophthalmic AI classifiers. Design: Comparison of calibration and classification performance among ophthalmic computer vision model training techniques.Subjects: 11,358 children aged 0-9 years evaluated as part of multinational trachoma screening, and 2,100 adult subjects with fungal and/or bacterial keratitis collated from multiple prior clinical studies. Methods:We compared latent supervision against supervised learning methods in two computer vision scenarios: trachoma screening with labels from different grader teams, and pathogen (fungal vs bacterial) differentiation in infectious keratitis using labels derived from culture and smear results.Main Outcome Measures: Classification performance was measured using the area under the receiver operating characteristic curve (AUROC), F1 score, and accuracy. Model calibration was assessed by calibration curves and Brier score decomposition.Results: For the trachoma screening scenario, the grader 1, grader 2, grader 3, and ensemble supervised models had AUROCs of 0.88, 0.91, 0.90 and 0.93, respectively.The latent supervision model had an AUROC of 0.94 with better calibration.For the infectious keratitis scenario, the culture-supervised model outperformed the smearsupervised model in bacterial keratitis classification (AUROC 0.87 vs 0.79), while the

Ähnliche Arbeiten