Deep learning-based multi-modal data integration enhancing breast cancer disease-free survival prediction

Abstract

Background: The prognosis of breast cancer is often unfavorable, emphasizing the need for early metastasis risk detection and accurate treatment predictions. This study aimed to develop a novel multi-modal deep learning model using preoperative data to predict disease-free survival (DFS).

Methods: We retrospectively collected pathology imaging, molecular and clinical data from The Cancer Genome Atlas and one independent institution in China. We developed a novel Deep Learning Clinical Medicine Based Pathological Gene Multi-modal (DeepClinMed-PGM) model for DFS prediction, integrating clinicopathological data with molecular insights. The patients included the training cohort (n = 741), internal validation cohort (n = 184), and external testing cohort (n = 95).

Result: Integrating multi-modal data into the DeepClinMed-PGM model significantly improved area under the receiver operating characteristic curve (AUC) values. In the training cohort, AUC values for 1-, 3-, and 5-year DFS predictions increased to 0.979, 0.957, and 0.871, while in the external testing cohort, the values reached 0.851, 0.878, and 0.938 for 1-, 2-, and 3-year DFS predictions, respectively. The DeepClinMed-PGM's robust discriminative capabilities were consistently evident across various cohorts, including the training cohort [hazard ratio (HR) 0.027, 95% confidence interval (CI) 0.0016-0.046, P < 0.0001], the internal validation cohort (HR 0.117, 95% CI 0.041-0.334, P < 0.0001), and the external cohort (HR 0.061, 95% CI 0.017-0.218, P < 0.0001). Additionally, the DeepClinMed-PGM model demonstrated C-index values of 0.925, 0.823, and 0.864 within the three cohorts, respectively.

Conclusion: This study introduces an approach to breast cancer prognosis, integrating imaging and molecular and clinical data for enhanced predictive accuracy, offering promise for personalized treatment strategies.

Keywords

breast cancer / multi-modality / deep learning / pathological / disease-free survival

Acknowledgements

This work was supported in part by the grants 2022B1212010006 and UICR0600008-6 from the Guangdong Provincial Key Laboratory IRADS, in part by the grants R0400001-22 and R0400025-21 from Guangdong Higher Education Upgrading Plan (2021-2025) of “Rushing to the Top, Making Up Shortcomings and Strengthening Special Features” with UIC research, grant 2023YFE0204000 from the National Key R&D Program of China, grants 2020A20070 and 2021AKP0003 from Macau Science and Technology Development Fund, Macao, grant 2023B1212060013 from the Science and Technology Planning Project of Guangdong Province, grant 82273204 from the National Natural Science Foundation of China, grants 2023A1515012412 and 2023A1515011214 from Guangdong Basic and Applied Basic Research Foundation, grants 2023A03J0722 and 202206010078 from the Guangzhou Science and Technology Project, grant 2018007 from the Sun Yat-Sen University Clinical Research 5010 Program, grant SYS-C-201801 from the Sun Yat-Sen Clinical Research Cultivating Program, grant A2020558 from the Guangdong Medical Science and Technology Program, grant 7670020025 from Tencent Charity Foundation, grants YXQH202209 and SYSQH-II-2024-07 from the Sun Yat-sen Pilot Scientific Research Fund, and grant 2023KQNCX138 from Guangdong Provincial Introduction of Innovative Research and Development Team.

Supplementary data

Supplementary data are available at PCMEDI online.

Conflict of interest

None declared.

Ethics statement

This retrospective, multicohort, diagnostic study was conducted according to the Declaration of Helsinki and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline. The study protocol was approved by the ethics committee of Sun Yat-sen Memorial Hospital of Sun Yat-sen University.

Data and code availability

All data reported in this paper will be shared by the lead contact upon request. This paper does not report original code. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

References

[1] Rebeca LS, Kimberly DM, Nikita SW et al. Cancer statistics 2023. CA Cancer J Clin 2023; 71:17-48. https://doi.org/10.3322/caac.21763.

[2] Loibl S, Poortmans P, Morrow M et al. Breast cancer. Lancet North Am Ed 2021;397:1750-69. https://doi.org/10.1016/S0140-6736(20)32381-3.

[3] Adrienne GW, Eric PW. Breast cancer treatment: A review. JAMA 2019;321:288-300. https://doi.org/10.1001/jama.2018.19323.

[4] Bera K, Schalper KA, Rimm DL et al. Artificial intelligence in digital pathology-new tools for diagnosis and precision oncology. Nat Rev Clin Oncol 2019;16:703-15. https://doi.org/10.1038/s41571-019-0252-y.

[5] Chen RJ, Lu MY, Wang J et al. Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans Med Imaging 2020;41:757-70. https://doi.org/10.1109/TMI.2020.3021387.

[6] Cheerla A, Gevaert O. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 2019;35:i446-54. https://doi.org/10.1093/bioinformatics/btz342.

[7] Haug CJ, Drazen JM. Artificial intelligence and machine learning in clinical medicine. N Engl J Med 2023;388:1201-8. https://doi.org/10.1056/NEJMra2302038.

[8] Swanson K, Wu E, Zhang A et al. From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment. Cell 2023;186:1772-91. https://doi.org/10.1016/j.cell.2023.01.035.

[9] Rajpurkar P, Chen E, Banerjee O et al. AI in health and medicine. Nat Med 2022;28:31-8. https://doi.org/10.1038/s41591-021-01614-0.

[10] Lu MY, Chen TY, Williamson DFK et al. AI-based pathology predicts origins for cancers of unknown primary. Nature 2021;594:106-10. https://doi.org/10.1038/s41586-021-03512-4.

[11] Steyaert S, Pizurica M, Nagaraj D et al. Multimodal data fusion for cancer biomarker discovery with deep learning. Nat Mach Intell 2023;5:351-62. https://doi.org/10.1038/s42256-023-00633-5.

[12] Zhou HY, Yu Y, Wang C et al. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nat Biomed Eng 2023;7:743-55. https://doi.org/10.1038/s41551-023-01045-x.

[13] Wang T, Gao Y. Metabolic insights into tumor pathogenesis: unveiling pan-cancer metabolism and the potential of untargeted metabolomics. MedComm Future Med. 2023;2:e59. https://doi.org/10.1002/mef2.59.

[14] Lu MY, Williamson DFK, Chen TY et al. Data-efficient and weakly supervised computational pathology on whole-slide images. Nat Biomed Eng 2021;5:555-70. https://doi.org/10.1038/s41551-020-00682-w.

[15] He K, Zhang X, Ren S et al. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016: 770-8.

[16] Cox DR. Regression models and life-tables (with discussion). Journal of the Royal Statistical Society Series B (Statistical Methodology) 1971;34:2517-6161.

[17] Liaw A, Wiener M. Classification and regression by randomForest. R News 2002;23:18-22.

[18] Aran D, Hu Z, Butte AJ. xCell: digitally portraying the tissue cellular heterogeneity landscape. Genome Biol 2017;18:220. https://doi.org/10.1186/s13059-017-1349-1.

[19] Ilse M, Tomczak JM, Welling M. Attention-based deep multiple instance learning. 2018; arXiv.1802.04712.

[20] Katzman JL, Shaham U, Cloninger A et al. Personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Med Res Methodol 2018;18:24. https://doi.org/10.1186/s12874-018-0482-1.

[21] Selvaraju RR, Das A, Vedantam R et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision. IEEE 2017: 618-26.

[22] Ashburner M, Ball CA, Blake JA et al. Gene ontology: tool for the unification of biology. The Gene Ontology Consortium. Nat Genet 2000;25:25-9. https://doi.org/10.1038/75556.

[23] Minoru K, Susumu G, Yoko S et al. KEGG for integration and interpretation of large-scale molecular data sets. Nucleic Acids Res 2012;40:D109-14. https://doi.org/10.1093/nar/gkr988.

[24] Kerr KF, Brown MD, Zhu K et al. Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use. JCO 2016;34:2534-40. https://doi.org/10.1200/JCO.2015.65.5654.

[25] Tinsley E, Bredin P, Toomey S et al. KMT2C and KMT2D aberrations in breast cancer. Trends Cancer 2024. https://www.cell.com/trends/cancer/abstract/S2405-8033(24)00029-3.

[26] Franco I, Alshalalfa M, Hernandez A et al. Genomic Characterization of Aggressive Breast Cancer in Younger Women. Ann Surg Oncol 2023;30:7569-78. https://doi.org/10.1245/s10434-023-14080-4.

[27] Park YH, Im SA, Park K et al. Longitudinal multi-omics study of palbociclib resistance in HR-positive/HER2-negative metastatic breast cancer. Genome Med 2023;15:55. https://doi.org/10.1186/ s13073-023-01201-7.

[28] Dawkins JB, Wang J, Maniati E et al. Reduced expression of histone methyltransferases KMT2C and KMT2D correlates with improved outcome in pancreatic ductal adenocarcinoma. Cancer Res 2016;76:4861-71. https://doi.org/10.1158/0008-5472.CAN-16-0481.

[29] Ko KP, Huang Y, Zhang S et al. Key genetic determinants driving esophageal squamous cell carcinoma initiation and immune evasion. Gastroenterology 2023;165:613-628.e20. https://doi.org/10.1053/j.gastro.2023.05.030.

[30] Mongan M, Wang J, Liu H et al. Loss of MAP3K1 enhances proliferation and apoptosis during retinal development. Development 2011;138:4001-12. https://doi.org/10.1242/dev.065003.

[31] Salem SM, Hamed AR, Mosaad RM. MTDH and MAP3K1 are direct targets of apoptosis-regulating miRNAs in colorectal carcinoma. Biomed Pharmacother 2017;94:767-73. https://doi.org/10.1016/j.biopha.2017.07.153.

[32] Yala A, Mikhael PG, Lehman C et al. Optimizing risk-based breast cancer screening policies with reinforcement learning. Nat Med 2022;28:136-43. https://doi.org/10.1038/s41591-021-01599-w.

[33] Dvijotham KD, Winkens J, Barsbey M et al. Enhancing the reliability and accuracy of AI-enabled diagnosis via complementaritydriven deferral to clinicians. Nat Med 2023;29:1814-20. https://doi.org/10.1038/s41591-023-02437-x.

[34] Boehm KM, Aherne EA, Ellenson L et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nat Cancer 2022;3:723-33. https://doi.org/10.1038/s43018-022-00388-9.

[35] Sammut SJ, Crispin-Ortuzar M, Chin SF et al. Multi-omic machine learning predictor of breast cancer therapy response. Nature 2022;601:623-9. https://doi.org/10.1038/s41586-021-04278-5.

[36] Singhal K, Azizi S, Tu T et al. Large language models encode clinical knowledge. Nature 2023;620:172-80. https://doi.org/10.1038/s41586-023-06291-2.

[37] Song AH, Jaume G, Williamson DFK et al. Artificial intelligence for digital and computational pathology. Nat Rev Bioeng 2023;1:930-49. https://doi.org/10.1038/s44222-023-00096-8.

[38] Zhang Q, Xu Y, Kang S et al. A novel computational framework for integrating multidimensional data to enhance accuracy in predicting the prognosis of colorectal cancer. MedComm Future Med 2022; 1:e27. https://doi.org/10.1002/mef2.27.

[39] Wang H, Fu T, Du Y et al. Scientific discovery in the age of artificial intelligence. Nature 2023;620:47-60. https://doi.org/10.1038/s41586-023-06221-2.

[40] Moor M, Banerjee O, Abad ZSH et al. Foundation models for generalist medical artificial intelligence. Nature 2023;616:25965. https://doi.org/10.1038/s41586-023-05881-4.

[41] Diao JA, Wang JK, Chui WF et al. Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes. Nat Commun 2021;12:1613. https://doi.org/10.1038/s41467-021-21896-9.

[42] Schmauch B, Romagnoni A, Pronier E et al. A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nat Commun 2020;11:3877. https://doi.org/10.1038/s41467-020-17678-4.