EACNet: Ensemble adversarial co-training neural network for handling missing modalities in MRI images for brain tumor segmentation

Abstract

Brain tumor segmentation is critical in clinical diagnosis and treatment planning. Existing methods for brain tumor segmentation with missing modalities often struggle when dealing with multiple missing modalities, a common scenario in real-world clinical settings. These methods primarily focus on handling a single missing modality at a time, making them insufficiently robust for the additional complexity encountered with incomplete data containing various missing modality combinations. Additionally, most existing methods rely on single models, which may limit their performance and increase the risk of overfitting the training data. This work proposes a novel method called the ensemble adversarial co-training neural network (EACNet) for accurate brain tumor segmentation from multi-modal magnetic resonance imaging (MRI) scans with multiple missing modalities. The proposed method consists of three key modules: the ensemble of pre-trained models, which captures diverse feature representations from the MRI data by employing an ensemble of pre-trained models; adversarial learning, which leverages a competitive training approach involving two models; a generator model, which creates realistic missing data, while sub-networks acting as discriminators learn to distinguish real data from the generated “fake” data. Co-training framework utilizes the information extracted by the multimodal path (trained on complete scans) to guide the learning process in the path handling missing modalities. The model potentially compensates for missing information through co-training interactions by exploiting the relationships between available modalities and the tumor segmentation task. EACNet was evaluated on the BraTS2018 and BraTS2020 challenge datasets and achieved state-of-the-art and competitive performance respectively. Notably, the segmentation results for the whole tumor (WT) dice similarity coefficient (DSC) reached 89.27%, surpassing the performance of existing methods. The analysis suggests that the ensemble approach offers potential benefits, and the adversarial co-training contributes to the increased robustness and accuracy of EACNet for brain tumor segmentation of MRI scans with missing modalities. The experimental results show that EACNet has promising results for the task of brain tumor segmentation of MRI scans with missing modalities and is a better candidate for real-world clinical applications.

Keywords

deep learning / magnetic resonance imaging (MRI) / medical image analysis / semantic segmentation / segmentation accuracy / image synthesis

References

[1] ILIC I, ILIC M. International patterns and trends in the brain cancer incidence and mortality: an observational study based on the global burden of disease. Heliyon, 2023, 9(7): e18222.

[2] OSTROM Q T, PATIL N, CIOFFI G, et al. CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2016-2020. Neuro-Oncology, 2023, 25(12): 90-99.

[3] OSTROM Q T, PRICE M, NEFF C, et al. CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2014-2018. Neuro-Oncology, 2021, 23(3): 61-105.

[4] MARTUCCI M, RUSSO R, SCHIMPERNA F, et al. Magnetic resonance imaging of primary adult brain tumors: state of the art and future perspectives. Biomedicines, 2023, 11(2): 3-64.

[5] ENDERS J, ZIMMERMANN E, RIEF M, et al. Reduction of claustrophobia during magnetic resonance imaging: methods and design of the “CLAUSTRO” randomized controlled trial. BMC Medical Imaging, 2011, 11(4): 4-10.

[6] KHANAL A, ESTRADA R. Dynamic deep networks for retinal vessel segmentation. Frontiers in Computer Science, 2020, 27(8): 51-64.

[7] ABDUSALOMOV A B, MUKHIDDINOV M, WHANGBO T K. Brain tumor detection based on deep learning approaches and magnetic resonance imaging. Cancers, 2023, 15(16): 41-72.

[8] WANG Y X, ZHANG Y, LIU Y, et al. ACN: adversarial co-training network for brain tumor segmentation with missing modalities//Medical Image Computing and Computer Assisted Intervention(MICCAI), September 27-October 1, 2021, Strasbourg: France. Cham: Springer International Publishing, 2021: 410-420.

[9] MA B T, WANG Y S, WANG S. Analyzing deep learning based brain tumor segmentation with missing MRI modalities. 2022: 2208.03470.

[10] TING H, LIU M. Multimodal transformer of incomplete MRI data for brain tumor segmentation. IEEE Journal of Biomedical and Health Informatics, 2024, 28 (1): 89-99.

[11] ZHOU T X, CANU S, VERA P, et al. Latent correlation representation learning for brain tumor segmentation with missing MRI modalities. IEEE Transactions on Image Processing, 2021, 30(1): 4263-4274.

[12] RAUT P, BALDINI G, SCHÖNECK M, et al. Using a generative adversarial network to generate synthetic MRI images for multi-class automatic segmentation of brain tumors. Frontiers in Radiology, 2024, 3(2): 1336902.

[13] SHEN Y, GAO M C. Brain tumor segmentation on MRI with missing modalities//The 26th Information Processing in Medical Imaging. June 2-7, 2019, Hong Kong, China. Cham: Springer International Publishing, 2019: 417-428.

[14] FENG X, TUSTISON N J, PATEL S H, et al. Brain tumor segmentation using an ensemble of 3D U-nets and overall survival prediction using radiomic features. Frontiers in Computational Neuroscience, 2020, 1(4): 1662.

[15] LIU S Y, WANG H R, LI S M, et al. Mixture-of-experts and semantic-guided network for brain tumor segmentation with missing MRI modalities. Medical & Biological Engineering & Computing, 2024, 62(10): 3179-3191.

[16] JEONG S W, CHO H H, LEE S, et al. Robust multimodal fusion network using adversarial learning for brain tumor grading. Computer Methods and Programs in Biomedicine, 2022, 226: 107165.

[17] LI Z Y, ZHANG Y F, LI H F, et al. Deformation-aware and reconstruction-driven multimodal representation learning for brain tumor segmentation with missing modalities. Biomedical Signal Processing and Control, 2024, 91: 106012.

[18] YUAN Y X, HUANG Y W, ZHOU Y. Rethinking a unified generative adversarial model for MRI modality completion//The 3rd MICCAI Workshop on Deep Generative Models. October 8, 2023, Vancouver, Canada.Cham: Springer Nature Switzerland, 2024: 143-153.

[19] CHEN H, QIN Z G, DING Y, et al. Brain Tumor Segmentation with Generative Adversarial Nets//2019 2nd International Conference on Artificial Intelligence and Big Data, May 25-28, 2019. Chengdu, China. New York: IEEE, 2019: 301-305.

[20] SAHOO S, MISHRA S, PANDA B, et al. An augmented modulated deep learning based intelligent predictive model for brain tumor detection using GAN ensemble. Sensors, 2023, 23(15): 6930.

[21] KALEJAHI B K, MESHGINI S, DANISHVAR S. Segmentation of brain tumor using a 3D generative adversarial network. Diagnostics, 2023, 13(21): 3344.

[22] LIU H, WEI D, LU D H, et al. M3AE: multimodal representation learning for brain tumor segmentation with missing modalities. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(2): 1657-1665.

[23] AZAD R, KHOSRAVI N, MERHOF D. SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities. International Conference on Medical Imaging with Deep Learning, July 6-8, 2022, Lürich, Switzerland. New York: IEEE, 2022, 1040(37): 1746-8094.

[24] WANG N, CHEN C W, MA L Z, et al. Latent feature regularization based adversarial network for brain tumor anomaly detection//2023 IEEE International Conference on Multimedia and Expo, July 10-14, 2023, Brisbane,Australia. New York: IEEE, 2023: 954-959.