The MAIDA initiative: establishing a framework for global medical-imaging data sharing

Abstract

A central question in developing artificial intelligence (AI) for the interpretation of medical images is whether these algorithms will work safely and effectively across diverse patient populations and clinical settings.1Rajpurkar P Lungren MP The current and future state of AI interpretation of medical images.N Engl J Med. 2023; 388: 1981-1990Crossref PubMed Scopus (22) Google Scholar Public datasets are the basis for training and validating AI models, making them essential for the rigorous assessment of performance and reliability that is required by regulatory bodies such as the US Food and Drug Administration.2Seastedt KP Schwab P O'Brien Z et al.Global healthcare fairness: we should be sharing more, not less, data.PLOS Digit Health. 2022; 1e0000102Crossref PubMed Google Scholar, 3Wu E Wu K Daneshjou R Ouyang D Ho DE Zou J How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals.Nat Med. 2021; 27: 582-584Crossref PubMed Scopus (166) Google Scholar However, current public datasets seldom have the diversity required to adequately evaluate algorithmic generalisability.4Kaushal A Altman R Langlotz C Geographic distribution of US cohorts used to train deep learning algorithms.JAMA. 2020; 324: 1212-1213Crossref PubMed Scopus (97) Google Scholar More comprehensive and varied datasets would improve the assessment of AI models and their ability to generalise across patient demographics, clinical environments, imaging equipment, and geographical regions. The scarcity of diverse public data also impedes optimal AI deployment strategies for specific medical settings. For example, there is the question of whether models should undergo so-called site-specific fine-tuning, which refers to the process of further training a pre-trained model on the local data of the target clinical site. This additional training can help avoid decreased model performance that could result from differences in patient population between the original model training data and the clinical site before approval for deployment in new hospitals. This issue has broad implications from immediate patient care to regulation. Comprehensive evaluations are needed to inform regulatory guidelines regarding dataset composition for model validation and requirements for local and regional adaptation.5Glocker B Robinson R Castro DC Dou Q Konukoglu E Machine learning with multi-site imaging data: an empirical study on the impact of scanner effects.arXiv. 2019; (published online Oct 10) (preprint).https://doi.org/10.48550/arXiv.1910.04597Google Scholar Here, we introduce the Medical AI Data for All (MAIDA) initiative, which is pioneering a framework for global medical-imaging data sharing to address the shortage of public health data and enable rigorous evaluation of AI models across all populations. Similar to its Hindi namesake of an essential flour, MAIDA aims to provide key ingredients to thoroughly assess AI through rich, diverse datasets. The initiative is in development with coordinating global partners while remaining locally adaptable to curate comprehensive, representative datasets. MAIDA is a collaborative effort in assembling diverse medical-imaging data at scale for rigorous AI assessments across diverse populations and settings. MAIDA has collaboratively engaged with a range of hospitals worldwide to curate and release a focused but diverse dataset. Our collection strategy was to acquire 100 medical scans per setting, a number that was a balance between obtaining a representative sample from each institution and managing logistical considerations. This sample size also enabled us to effectively test the robustness of our models in distribution shifts. For outreach, we engaged with individual researchers and radiology departments worldwide. Furthermore, we attracted interest from individuals who learnt about MAIDA in various ways, including our website, social media, or conference presentations. We refer to these individuals, with whom we directly collaborate at their institutions, as local champions. Although most participants were doctors, we also welcomed computer scientists who were primarily affiliated with large academic medical centres. We discovered that clinical knowledge was invaluable for maintaining data quality. To inform all partners of our project goals, we conducted 30-min meetings in which we outlined the project and offered standard templates for institutional review boards (IRBs) and data transfer and use agreements (DTUAs) to simplify compliance (figure). Despite our efforts, approval protocols differed among institutions. We used a US Federal Demonstration Partnership (FDP) template to standardise data sharing. Challenges emerged, including institutions that were unwilling to publicly share data and that required modifications to DTUAs, which resulted in delays. Local champions in these organisations were often pivotal in advancing the process. Although the majority of institutions accepted our standard templates, the process taught us to be flexible and transparent in our communication and to consider institutional differences in approval timelines and data-sharing preferences. We provided comprehensive guidelines to our partners regarding data collection and de-identification. The documentation delineated detailed inclusion and exclusion criteria and offered a manual for downloading data-recording worksheets, randomising selections, and collecting data samples that met our stipulations. Despite the detailed guidelines, not all partners fully engaged with the written documentation. To address this issue, meetings were offered to review the protocols in detail. Furthermore, medical practices varied between institutions, meaning our standard guidelines were not universally applicable. For example, hospitals in countries other than the USA rarely collect data on the race of patients, some issue chest x-rays to all patients indiscriminately, and radiology reports might be absent in some settings.6Pinto AD Eissa A Kiran T Mashford-Pringle A Needham A Dhalla I Considerations for collecting data on race and Indigenous identity during health card renewal across Canadian jurisdictions.CMAJ. 2023; 195: E880-E882Crossref Scopus (0) Google Scholar, 7Burute N Jankharia B Teleradiology: the Indian perspective.Indian J Radiol Imaging. 2009; 19: 16-18Crossref PubMed Scopus (0) Google Scholar Early detection of these issues allowed us to adapt solutions to individual cases during instructional meetings. To mitigate the risk of exposure of protected health information (PHI) during data transmission, partners were required to complete data de-identification before data sharing. We provided specific guidelines that were adapted to different data types. For structured data, our data-recording worksheets were designed to omit variables that were classified as PHI and partners were instructed to modify any potentially identifying data. For free-form radiology reports, partners were asked to manually remove any identifiers, such as names and dates. For images, the guidelines specified the exclusion of images displaying jaws with teeth, partial skulls, or jewellery. Furthermore, we developed a tool to enhance the de-identification of digital imaging and communications in medicine (DICOM) files, which commonly contain extensive metadata—some of which might be PHI. To preclude inadvertent disclosure of PHI, our tool extracted pixel values and non-PHI metadata and saved them separately as PNG and CSV files. Partners only shared files with non-PHI metadata, thereby eliminating the risk of PHI exposure through embedded DICOM metadata. This tool permitted hospital security teams to review both the content and the operational logic via Python scripts, if required. Before any data sharing, in-person meetings were conducted with each partner to validate the correct execution of data-collection and data-de-identification procedures. Upon receipt of data, a meticulous review was conducted by our research team to confirm the absence of any identifiers and assess the completeness of the dataset. Substantial time was invested in liaising with our partners. Despite our efforts to streamline the MAIDA process by offering IRB and DTUA templates, as well as limiting data requests to 100 patient records per clinical setting, the timeframe for completion was often several months. Most delays were attributable to waiting for IRB approvals and DTUA signatures, especially from institutions with monthly ethics-committee meetings or complex legal-review mechanisms. According to feedback from multiple partners, the actual tasks of data collection and de-identification could typically be accomplished within 1 week. Chest x-rays are the most widely used radiological tests worldwide, yet their interpretation is prone to error.8Gefter WB Post BA Hatabu H Commonly missed findings on chest radiographs: causes and consequences.Chest. 2023; 163: 650-661Summary Full Text Full Text PDF PubMed Scopus (0) Google Scholar Despite the substantial focus in AI research, the generalisability of existing models across diverse clinical settings remains insufficiently evaluated as current public datasets are insufficient in size, diversity, or scope or might not have reliable annotations (appendix). The MAIDA initiative aims to improve both the quality and the breadth of the interpretation of chest x-rays in three key clinical settings: the intensive care unit (ICU), the neonatal ICU, and the emergency department. In the adult ICU, MAIDA focuses on automating endotracheal-tube assessments to mitigate frequent misplacements and severe complications. In the neonatal ICU, the initiative aims for precise endotracheal-tube placements, considering the minimal error margin in this vulnerable group. In the emergency department, MAIDA targets quick and consistent pneumonia detection through machine learning to improve efficiency via collaborations between clinicians and AI.9Agarwal N Moehring A Rajpurkar P Salz T Combining human expertise with artificial intelligence: experimental evidence from radiology.https://www.nber.org/system/files/working_papers/w31422/w31422.pdfDate: 2023Date accessed: November 9, 2023Google Scholar To assemble a comprehensive and reliable dataset, MAIDA collects images and metadata that is extracted from DICOM files. This dataset includes the reason for the chest x-ray, available radiology reports, and demographic details (eg, age, race, and sex). For a nuanced understanding, clinically relevant data, such as vital signs and specialised test results, are also included if available. Through the MAIDA initiative, we plan to progressively release datasets that have been collected across diverse global regions and clinical environments. The first dataset release is expected in early 2024, with additional datasets as partnerships expand. We aim to make these datasets publicly available to enable open research into assessing and improving AI models for medical imaging. On the basis of insights that have been gained through the MAIDA initiative, we suggest developing comprehensive IRB protocols and data-sharing agreements in advance. Engaging legal teams early on might help to streamline administrative processes. Anticipating the adequate time to secure IRB approvals and finalise data-sharing contracts can be prudent, as these processes often require substantial time. Maintaining rigorous documentation that delineates inclusion and exclusion criteria and that emphasises the removal of PHI might promote dataset consistency while minimising potential confounders in subsequent analyses. However, recognising that medical practices vary across institutions and being flexible when applying standard protocols could prove beneficial. Thus, proactively working with partners to address unique challenges might be advisable. Direct interactions through in-person meetings can be useful to address questions and show proper data-collection and data-de-identification techniques, especially for partners who are less acquainted with the complexities of medical data sharing. We declare no competing interests. We thank Wendy Erselius (Department of Biomedical Informatics, Harvard Medical School, Harvard University, Boston, MA, USA) for their instrumental role in managing partnerships and coordinating logistics to establish a diverse coalition of hospitals for the Medical AI Data for All initiative (MAIDA) initiative. We thank Cassandra Perry and Jennifer Sullivan (Department of Biomedical Informatics, Harvard Medical School, Harvard University, Boston, MA, USA) for helping to develop agreements that enabled secure data sharing across institutions. We thank the dedicated individuals partnering in the MAIDA initiative, working to assemble diverse medical datasets, and making these datasets available for research that can benefit people worldwide. Download .pdf (.15 MB) Help with pdf files Supplementary appendix

Ähnliche Arbeiten