The need for a prediction model assessment framework

Abstract

We thank Mohammad Jalali and colleagues (December, 2020)1Jalali MS DiGennaro C Sridhar D Transparency assessment of COVID-19 models.Lancet Glob Health. 2020; 8: e1459-e1460Summary Full Text Full Text PDF PubMed Scopus (19) Google Scholar for highlighting the need for transparency assessments of COVID-19 models. The authors evaluated the transparency of COVID-19 models using a 27-point binary criterion adopted with the use of three different checklists, and reported that more than half of the studies did not share their longitudinal data, and only 14% of the studies met 90% of the transparency items on their checklist. However, the authors did not consider the MINimum Information for Medical artificial intelligence (AI) Reporting (MINIMAR),2Hernandez-Boussard T Bozkurt S Ioannidis JPA Shah NH MINIMAR (MINimum Information for Medical AI Reporting): developing reporting standards for artificial intelligence in health care.J Am Med Inform Assoc. 2020; 27: 2011-2015Crossref PubMed Scopus (110) Google Scholar which provides reporting standards for AI model projections in health care. Furthermore, the checklists they used are not specific for assessing the transparency of prediction models and it is not clear how these criteria were applied. Jalali and colleagues also argue that model developers are largely responsible for providing transparency and not journals, which we do not fully agree on, because journals also have an obligation to hold authors responsible when it comes to providing details of key materials and information.3Haibe-Kains B Adam GA Hosny A et al.Transparency and reproducibility in artificial intelligence.Nature. 2020; 586: E14-E16Crossref PubMed Scopus (172) Google Scholar The recently released Consolidated Standards of Reporting Trials (CONSORT)-AI and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT)-AI guidelines provide the reporting standards for clinical trials that use AI.4Liu X Cruz Rivera S Moher D Calvert MJ Denniston AK Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension.Nat Med. 2020; 26: 1364-1374Crossref PubMed Scopus (261) Google Scholar The plethora of guidelines highlights the need for a consensus on a framework for prediction model assessment with tangible indicators for model evaluation and reporting. In addition to a transparency assessment, prediction models without external validation and a prospective assessment of the net use of the models for reproducibility undermines its scientific value.5Jin J Agarwala N Kundu P Wang Y Zhao R Chatterjee N Transparency, reproducibility, and validity of COVID-19 projection models. Johns Hopkins Bloomberg School of Public Health.https://www.jhsph.edu/covid-19/articles/transparency-reproducibility-and-validation-of-covid-19-projection-models.htmlDate: June 22, 2020Date accessed: January 27, 2021Google Scholar Before a prediction model is used for public health decision making, it should be evaluated for its real-world performance through short-term and continuous model training and optimisation to avoid a distributional shift, such as model degradation because of a change in testing data. Moreover, it is essential to understand the model choices, amounts of complexity, and assumptions, including how the model accounted for various sources of uncertainty, namely how physical distancing, mask usage, and other covariates are defined and measured.6Garnett GP Cousens S Hallett TB Steketee R Walker N Mathematical models in the evaluation of health programmes.Lancet. 2011; 378: 515-525Summary Full Text Full Text PDF PubMed Scopus (166) Google Scholar Further, it is necessary to quantify socioeconomic factors, population behaviours, and government actions taken and planned into estimating the model projections with details of analysis. Along with the proper documentation, the projection models should share codes, software dependencies, and datasets via open-source frameworks, namely GitHub, Code Ocean, and ModelhHub resources. Also, it is crucial to consider the different types of bias in data and modelling, namely, measurement bias, evaluation bias, and deployment bias. COVID-19 projection models have been used widely for making public health planning and resource allocations. A scientifically validated COVID-19 project model might help health-care policy makers to better prepare for mitigating the effects of the pandemic, make informed decisions, and enact appropriate actions to save human lives. However, caution is needed to interpret these models. Otherwise, it could lead to an over-allocation or under-allocation of health-care resources, unnecessary suffering, and a mistrust in models. COVID-19 projection models have not yet reported the details of the data used for the development, training, and evaluation of the models, making it difficult to assess the bias and fairness of the model and its applicability. Recognising these limitations, there is a crucial need to develop a framework that includes transparency, reproducibility, and a prospective validation to evaluate COVID-19 projection models. A multidisciplinary task force including experts in infectious disease modelling, health informatics, data science, computer science, epidemiology, statistics, health-care administration, and policy making is needed to create a set of benchmark metrics for health-care model evaluation. SMSI reports grants from the National Health and Medical Research Council and the National Heart Foundation of Australia. All other authors declare no competing interests. Transparency assessment of COVID-19 modelsThe COVID-19 pandemic has strained societal structures and created a global crisis. Scientific models have a crucial role in mitigating harm from the pandemic, by estimating the spread of outbreaks of the virus and analysing the effects of public health policies. The context-sensitive and time-sensitive measures provided by COVID-19 models offer real population health impacts and are of great importance. However, these models must be completely transparent before policies and insights are enacted. Full-Text PDF Open AccessThe need for a prediction model assessment framework – Authors' replyWe appreciate the feedback provided by Islam and Khosravi on our assessment.1 We agree that journals play an important role in prioritising transparency and reproducibility in publication decisions. Transparency guidelines have been developed. More than 1100 journal publishers have adopted the Transparency and Openness Promotion guidelines,2 including several journals that published models evaluated in our analysis. Despite these important initiatives and ongoing improvements, journals' peer review processes do not fully assess and control for model transparency. Full-Text PDF Open Access

Ähnliche Arbeiten