Multiple External Validation of Clinical Prediction Rules (CPRs), Adjusting CPRs and Individual Patient Data meta-analysis: methodological challenges

Abstract

Introduction The most useful CPRs are those that can be applied across multiple settings, have clinical validity and improve patient outcome. In a previous study we systematically identified CPRs to detect serious infection (SI) in acutely ill children. (e.g. Yale Observation Scale (YOS), a 5-stage decision tree) First, we verified the CPRs’ validity across an international dataset network. Secondly we investigated the value of adding vital signs to improve CPRs, through individual patient data (IPD) meta-analysis and the inherent methodological challenges. Method We used 7 datasets (11,045 children) to validate the CPRs. When positive, CPRs can substantially raise the probability of illness if the positive likelihood ratio is more than 5.0, or, when negative, lower the probability if the negative likelihood ratio is less than 0.2. To improve a CPR by adjusting its components, the effects on validity need to be re-assessed. First we pool all data of the existing datasets to perform an IPD meta-analysis to predict the likelihood of SI. The challenges are: dealing with missing values, sensitivity and specificity correlations, defining methodology of the pooling and rule development, and adjusting for multicenter-aspect. Finally the resulting CPR will be ready for validation and impact analysis. Results The YOS provided no rule-out value (LR- ranging from 0.46 to 0.97). A 5-stage decision tree provided minimal ruling in value (LR+ from 0.88 to 1.64), LR- was above 0.2 in all datasets except one low prevalence validation-dataset (LR- of 0.11). Referral pattern did not influence the likelihood ratios. Conclusions The YOS did not meet our rule-out criteria in any dataset. The 5-stage decision rule met our criteria in one data set. CPRs offer different diagnostic value, depending on setting and prevalence. Heterogeneity can explain the reduced accuracy, through differences in definition of predictors, the case mix, and random error. Reproducibility and transportability should be assessed to ensure generalizability. Validating existing rules is urgently needed in order to better prepare their impact on clinical practice. If the new rule is externally validated in an independent population, IPD meta-analysis can improve the CPR development and validation process through assessment of the methodological challenges.

Ähnliche Arbeiten