Integrating radiology and histology via co-attention deep learning for predicting progression-free survival in patients with metastatic prostate cancer

Abstract

To the Editor: In the evolving landscape of prostate cancer management, particularly metastatic prostate cancer (mPCa), the transition to castration-resistant prostate cancer (CRPC) represents a pivotal challenge.[1] This progression marks a critical phase in which the disease prognosis typically deteriorates and therapeutic options become increasingly limited. Traditional prognostic models heavily depend on clinical indicators such as Prostate Imaging Reporting and Data System (PI-RADS) scores, prostate-specific antigen (PSA) levels, and Gleason scores.[2] While these markers are invaluable in the diagnostic context, their ability to predict progression to CRPC is notably limited. Furthermore, these prognostic markers do not fully capture the complex biological interactions within tumors or the dynamic nature of cancer progression. Current methods assess these indicators in isolation and thus frequently underestimate the aggressive nature of certain tumors or miss subtle cues that signify imminent progression to CRPC.[3,4] The limitations of existing prognostic approaches highlight the urgent need for more sophisticated predictive models that can integrate and interpret the vast and complex data available from different diagnostic modalities. For example, radiological images and pathological images provide valuable insights into tumor morphology and cellular composition. However, each modality offers a distinct perspective that, if effectively integrated, could provide a more comprehensive understanding of tumor behavior. Advances in machine learning, particularly deep learning, offer a promising avenue for harnessing these multimodal data.[5,6] By employing deep learning-based algorithms that are capable of detecting patterns and associations across different types of data, researchers can develop models that not only improve the accuracy of cancer progression predictions but also offer insights into the underlying mechanisms of resistance to standard therapies. In this study, we propose a cross-modality co-attention deep learning model to integrate radiological and pathological images for predicting progression-free survival in patients with mPCa. The cross-modality co-attention mechanism is designed to synergistically align and enhance the interpretability of each modality, thereby providing a unified view of the tumor characteristics. By integrating radiological and pathological images, this approach reveals interaction effects that are not observable when modalities are analyzed in isolation. This method of enhancing the prediction of survival among CRPC patients is crucial for extending the clinical management window during which effective treatments can be administered, ultimately aiming to improve outcomes in patients with mPCa. In this retrospective study, 366 patients diagnosed with mPCa across three hospitals were included. All procedures and methodological approaches were approved by the institutional review boards of the participating hospitals. Due to the retrospective design and the use of preexisting diagnostic data, the need for informed consent was waived under the ethical guidelines for retrospective studies, thereby adhering to all ethical standards for medical research. Patient selection was guided by specific inclusion and exclusion criteria. The inclusion criteria were as follows: pathologically confirmed mPCa; preoperative magnetic resonance imaging (MRI) and postoperative whole-slide imaging (WSI) data; and comprehensive clinicopathological information, including age, PSA levels, Gleason score, PI-RADS score, and clinical tumor-node-metastasis (TNM) staging. The exclusion criteria were as follows: other cancer types; prior treatments affecting PCa; incomplete clinical profiles; or poor-quality imaging data. This stringent selection process ensured the high quality and relevance of the data used for model training and testing. This patient cohort was systematically divided into distinct groups: a training cohort (n = 217), an internal validation cohort (IVC) (n = 54), external testing cohort 1 (n = 44), and external testing cohort 2 (n = 51). This group assignment facilitated a thorough evaluation of the predictive model across varying clinical environments. CRPC-free survival (CFS) was calculated as the interval from the initial treatment to the onset of CRPC or the end of follow-up. Patients who remained non-castration-resistant at the final follow-up or those who were lost to follow-up were considered censored. For radiological data, MR images, including T2-weighted and apparent diffusion coefficient (ADC) sequences, were collected. These images were processed via a pretrained ResNet-34 network adapted to extract meaningful features specific to prostate cancer. The preprocessing steps included resampling all the images to a standardized resolution to maintain consistency across the datasets from different institutions. Experienced radiologists manually delineated the tumor regions, with a senior radiologist reviewing these segmentations to resolve any discrepancies. Histological data were processed through WSIs, which were scanned and annotated by pathologists to identify cancerous regions. These annotated WSIs were then segmented into smaller patches at a 20× resolution, which were resized to dimensions of 224 × 224 pixels for further analysis.[7] The process of manual annotation was meticulous, ensuring that only regions with substantial cancerous presence were included, thus enhancing the specificity of the histological data input into the model. The core methodological innovation of this study lies in the feature extraction and integration process, facilitated by a deep learning framework. For each modality–radiology and histology–specific features were extracted via the pretrained ResNet-34 network. These modality-specific features were then integrated via a novel cross-modality co-attention mechanism designed to align and synthesize information across the radiological and histological modalities. In the proposed co-attention module, each 512-dimensional feature embedding extracted from the MR or WSI is regarded as a representation of the intramodality patterns and is used as the input of the co-attention module. A transformer-based attention block was used to capture the interconnections between modality-specific patterns, where the radiological embeddings added with class tokens were used as queries. The addition of radiological information to global histological information could enhance cross-modality interactions. The co-attention learning module is a critical component of our methodology, thus enabling the model to dynamically attend to and integrate complementary diagnostic information from both MRI and histology. This approach is designed to capitalize on the intrinsic strengths of each data type, potentially uncovering latent biomarkers and improving the model’s overall prognostic accuracy. The architecture of the deep learning model incorporates dual pathways for handling modality-specific data, converging into a co-attention mechanism that facilitates the synthesis of a unified feature set representing both radiological and histological insights. This setup allows the model to leverage complex interdependencies between the data types, enhancing its ability to predict disease progression. For validation, the model was first trained on the training cohort, tuned and validated internally, and then tested on two independent external cohorts to assess its generalizability and robustness. Statistical analyses, including multivariate Cox regression, prediction error curves over time, and time-dependent receiver operating characteristic (ROC) analysis, were conducted to evaluate the model’s performance, compare it against unimodal approaches, and confirm the independent prognostic value of the integrated risk score. The integrated deep learning model developed in this study demonstrated substantial efficacy in predicting progression to CRPC across diverse patient cohorts, highlighting its potential as a superior prognostic tool over traditional single-modality models. The model was evaluated on the basis of its concordance index (C-index), hazard ratio (HR) from Cox regression analysis, and area under the receiver operating characteristic curve (AUC), among other metrics. The C-index for the IVC reached 0.825, indicating excellent predictive ability, which slightly diminished in the external testing cohorts, with scores of 0.803 and 0.785, respectively. These results not only reinforce the consistency and reliability of the integrated model across different clinical settings but also illustrate its advantage over conventional methods, which predominantly rely on either radiological or histological data alone. One of the most significant findings from the study was the independent prognostic value of the integrated risk score derived from the model. Multivariate Cox regression analyses revealed the strong predictive power of the integrated risk score for CRPC progression, with HRs indicating a high level of statistical significance: HR = 11.109, with a 95% confidence interval (CI) of 3.342–36.933 in the IVC; HR = 33.075 (95% CI: 6.556–166.852) in the external testing cohort 1; and HR = 21.236 (CI: 5.103–88.602) in the external testing cohort 2. These values highlight the robustness of the risk score as an independent marker and suggest that it can significantly aid in identifying patients at increased risks for progressing to CRPC. Compared with the unimodal radiological model and unimodal pathological model, the integrated model demonstrated superior performance. The prediction error curves over time for all three models across these validation and external testing cohorts are shown in Supplementary Figure 1A–C, https://links.lww.com/CM9/C586. The average error for the integrated model across all patients was lower than that observed for the unimodal models. The time-dependent ROC curves with corresponding AUCs at 12 months, 24 months, and 36 months for the integrated model are shown in Supplementary Figure 1D–F, https://links.lww.com/CM9/C586. The AUCs for the integrated model ranged between 0.711 and 0.933. These results confirmed that the integrated model achieved higher prediction accuracy and better discriminative ability than both unimodal models did. While the current results are promising, the study also opens avenues for future research. Further investigations are needed to explore the model’s applicability in real-world clinical settings, including prospective trials, to validate its effectiveness in live clinical environments. Additionally, integrating other forms of data, such as genomic or biochemical markers, could enhance the model’s predictive capacity, thereby offering even more detailed insights into the biological underpinnings of mPCa progression. In conclusion, the study has been demonstrated that the integration of radiological and histological data through advanced machine learning techniques leads to significant advancements in predicting CRPC progression, marking a critical step forward in the management and treatment of mPCa. As such, this model not only contributes to the scientific understanding of cancer progression but also holds substantial potential for improving patient outcomes through enhanced prognostic precision. Funding This work was supported by grants from the Guangdong Basic and Applied Basic Research Foundation (Nos. 2024A1515030163 and 2025A1515011459), the Guizhou Senior Innovative Talent Project (No. QKHPTRC-GCC[2022]041-1), the National Natural Science Foundation of China (Nos. 12126608 and 82427809), the R&D project of Pazhou Lab (Huangpu) (No. 2023K0603), National Key Research and Development Program of China (No. 2023YFF0715400), Shenzhen Science and Technology Program (No. JCYJ20241202125014018), the Shenzhen Key Technology Research and Development (R&D) Project (No. JSGG20211029102001001), and Shenzhen Medical Research Fund (No. A2303008). Conflicts of interest None.

Ähnliche Arbeiten