From models to tools: clinical translation of machine learning studies in psychosis

Abstract

The past decade has seen a proliferation of neuroimaging-based machine learning studies in psychosis.1 Furthermore, within the span of ten years, small local studies with few dozen participants have evolved into large multi-centre studies with several hundreds of participants.2–5 In the midst of the search for accurate models, much attention has been given to methodological challenges including the impact of sample size,6,7 the limitations of traditional case–control designs,8,9 how to best deal with confounding variables10 and the effects of heterogeneity11,12 and inter-scanner variability,13 just to mention a few. Although there are still important methodological challenges to overcome, substantial progress is being made, and a solution to these challenges is now considered to be a matter of when rather than if.14,15 Wider discussions in the medical community about the ethical and legal implications of integrating machine learning models within diagnostic and prognostic assessment of patients are also underway.16–20 Taken collectively, the progress being made towards the development and validation of neuroimaging-based machine learning models is encouraging, as if the different pieces of a very complex puzzle were slowly coming together. Less discussed however, are the challenges related to the development and validation of machine learning-based clinical tools. Here the critical distinction is between “models”, which tend to be developed and validated using a limited number of well characterised datasets with the aim of maximising accuracy, sensitivity and specificity, and “tools”, which must be feasible, acceptable and safe, and provide information that will guide clinical decision-making in real-world settings. This is a timely discussion, as a new generation of multi-centre studies aiming to develop machine learning tools to manage patients with psychosis is emerging (e.g., PSYSCAN,21 PRONIA—www.pronia.eu). Let’s imagine that we have developed a neuroimaging-based machine learning model with high levels of accuracy, sensitivity and specificity, after addressing the main methodological issues.2 Next we’d like to translate this machine learning model into an actual clinical tool to support the assessment of individual patients. What are our main challenges along this translation? In this opinion piece, we discuss seven critical aspects that require careful consideration when moving from a “model” to a “tool”. These include real-world validation, clinical utility, feasibility, acceptability, safety and finally, dissemination. Real-world validation After validating our model using several independent datasets, collected using different scanners across multiple research sites, we might feel reassured about its performance in a real-world setting. Yet our optimism might be premature. This is because datasets collected for the purpose of research tend to include patients who meet stringent inclusion/exclusion criteria; unfortunately this highly selected group of patients differ from service users who do not take part in research (e.g., less severe, lower comorbidities, less medicated, and higher functioning).22 Therefore, when it comes to clinical validation, we need to consider not only the size but also the type of sample. In practice, the validation of a clinical tool should be done in a naturalist design, where all service users that may benefit from the tool are approached whilst having minimal exclusion criteria. It is likely that this will result in lower accuracies, sensitivities and specificities than the original validation using research datasets. The silver-lining here is that, if permitted, the more “naturalistic” data could be used to improve our tool. Learning from experience is, after all, one of the essential properties of machine learning.

Ähnliche Arbeiten