Quantification of BERT Diagnosis Generalizability Across Medical\n Specialties Using Semantic Dataset Distance

Abstract

Deep learning models in healthcare may fail to generalize on data from unseen\ncorpora. Additionally, no quantitative metric exists to tell how existing\nmodels will perform on new data. Previous studies demonstrated that NLP models\nof medical notes generalize variably between institutions, but ignored other\nlevels of healthcare organization. We measured SciBERT diagnosis sentiment\nclassifier generalizability between medical specialties using EHR sentences\nfrom MIMIC-III. Models trained on one specialty performed better on internal\ntest sets than mixed or external test sets (mean AUCs 0.92, 0.87, and 0.83,\nrespectively; p = 0.016). When models are trained on more specialties, they\nhave better test performances (p < 1e-4). Model performance on new corpora is\ndirectly correlated to the similarity between train and test sentence content\n(p < 1e-4). Future studies should assess additional axes of generalization to\nensure deep learning models fulfil their intended purpose across institutions,\nspecialties, and practices.\n

Ähnliche Arbeiten