Supervised Multimodal Fission Learning

Abstract

Learning from multimodal data sets can leverage complementary information and lead to improved performance for prediction tasks. A commonly used strategy to account for feature correlations in high-dimensional data sets is the latent variable approach. Several latent variable methods have been proposed for multimodal data sets; however, these methods either focus on extracting a shared component across all modalities or extracting a shared component and individual components specific to each modality, overlooking correlations within partial subsets of modalities. We propose multimodal fission learning (MMFL), the first supervised latent variable model that adopts a generalizable decomposition into globally joint, partially joint, and individual components from multimodal data sets. A key strength of MMFL is a natural extension to incorporate incomplete multimodal data in either training and test phases by leveraging the learned modality structure. Through simulation studies, we demonstrate that MMFL outperforms a variety of existing multimodal algorithms under both complete modality and incomplete modality settings. We applied MMFL to two real-world case studies: early prediction of Alzheimer’s disease using neuroimaging and genetic data and predicting posttraumatic headache improvement using clinical data collected via questionnaires and brain neuroimaging data. MMFL achieved improved predictive accuracy and enhanced interpretability offering insights for within- and cross-modal relationships of multimodal data sets. History: Rema Padman served as the senior editor for this article. Funding: This research was supported by the National Institutes of Health [Grants 2R42AG053149-02A1, R01AG069453, 30AG072980, R01AG069453, and 1R61NS113315-01], the National Science Foundation [Grant DMS-2053170], and the Department of Defense [Award W81XWH-19-0534]. The grantee organization is the Northern California Institute for Research and Education. In the past, ADNI has also received funding from the National Institute of Biomedical Imaging and Bioengineering, the Canadian Institutes of Health Research, and private sector contributions through the Foundation for the National Institutes of Health including generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. Data Ethics & Reproducibility Note: The code capsule is available at https://github.com/lingchm/MMFL and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2024.0059 ).

Ähnliche Arbeiten