CT-γ-Net: A Hybrid Model Based on Convolutional Encoder-Decoder and Transformer Encoder for Brain Tumor Localization

Abstract

Brain tumor is a life-threatening disease, and its early diagnosis can save human life. Computer-aided brain tumor segmentation and localization in magnetic resonance imaging (MRI) images have emerged as pivotal approaches for expediting the disease diagnosis process. In the past few decades, various researchers combined the strengths of convolutional networks and transformer to perform brain tumor segmentation. However, these models require a large number of trainable weights parameters, and there is still scope for performance improvement in them. To bridge these research gaps, this paper proposes a novel hybrid model named “CT-γ-Net” for effective and efficient brain tumor localization. The proposed CT-γ-Net model follows an encoder-decoder structure in which the convolutional encoder (CE) and transformer encoder (TE) are used for encoding, whereas the convolutional decoder (CD) is utilized for decoding the combined output of CE and TE to generate the segmentation masks. In CE and CD components of the CT-γ-Net model, conventional convolutional layers are replaced by depth-wise separable convolutional layers, as these layers significantly reduce trainable weights parameters. The proposed model achieves 95.5% MeanIoU, 94.82% Dice score, and 99.24% pixel accuracy on a publicly available dataset named the Cancer Imaging Archive. These experimental results demonstrate that the CT-γ-Net model outperformed other state-of-the-art research works, despite using roughly 28%fewer trainable weights parameters. Hence, the proposed model’s lightweight nature and its high performance make it a suitable candidate for deployment on mobile devices, facilitating the precise localization of brain tumor regions in MRI images.

Keywords

brain tumor segmentation / transformer / convolutional encoder-decoder / deep learning / disease diagnosis using artificial intelligence

Conflicts of Interest

The authors declare that they have no conflicts of interest to this work.

Data Availability Statement

The data that support the findings of this study are openly available in Kaggle at https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation.

Author Contribution Statement

Punam Bedi: Conceptualization, Validation, Writing - review & editing, Supervision, Project administration. Ningyao Ningshen: Conceptualization, Methodology, Software, Formal analysis, Writing -original draft, Visualization. Surbhi Rani: Conceptualization, Methodology, Software, Formal analysis, Writing - original draft, Visualization. Pushkar Gole: Conceptualization, Validation, Writing -review & editing. Veenu Bhasin: Conceptualization, Validation, Writing - review & editing, Supervision.

References

[1] Business Insider. (2023). Brain tumor cases rising “steadily” in India, 20% are children:Doctors. Retrieved from: https://www.businessinsider.in/science/health/news/brain-tumour-cases-rising-steadily-in-india-20-are-children-doctors/articleshow/100848073.cms

[2] Cancer. Net. (2024). Brain and spinal cord tumor in adults early detection, diagnosis, and staging. Retrieved from: https://www.cancer.net/cancer-types/brain-tumor/diagnosis

[3] Coupet M., Urruty T., Leelanupab T., Naudin M., Bourdon P., Maloigne C. F., & Guillevin R. (2022). A multi-sequences MRI deep framework study applied to glioma classification. Multimedia Tools and Applications, 81(10), 13563-13591. https://doi.org/10.1007/S11042-022-12316-1

[4] Hassan S., Hassan A. A., Marshad I., Al Hosain M. A., Amin M., Faisal F., & Nishat M. M. (2022). Comparative analysis of machine learning algorithms in detection of brain tumor. In 3rd International Conference on Big Data Analytics and Practices,31-36. https://doi.org/10.1109/IBDAP55587.2022.9907433

[5] Rinesh S., Maheswari K., Arthi B., Sherubha P., Vijay A., Sridhar S., :::, & Waji Y. A. (2022). Investigations on brain tumor classification using hybrid machine learning algorithms. Journal of Healthcare Engineering, 2022(1), 2761847. https://doi.org/10.1155/2022/2761847

[6] Bedi P., Ningshen N., Rani S., & Gole P. (2024). Explainable predictions for brain tumor diagnosis using InceptionV3 CNN architecture. In International Conference on Innovative Computing and Communications: Proceedings of ICICC 2023, 2,125-134. https://doi.org/10.1007/978-981-99-4071-4_11

[7] Corso J. J., Sharon E., Dube S., El-Saden S., Sinha U., & Yuille A. (2008). Efficient multilevel brain tumor segment-ation with integrated Bayesian model classification. IEEE Transactions on Medical Imaging, 27(5), 629-640. https://doi.org/10.1109/TMI.2007.912817

[8] Meier R., Bauer S., Slotboom J., Wiest R., & Reyes M.(2014). Appearance-and context-sensitive features for brain tumor segmentation. In Proceedings of MICCAI BRATS Challenge. https://doi.org/10.13140/2.1.3766.7846

[9] Pei L., Reza S. M. S., Li W., Davatzikos C., & Iftekharuddin K. M. (2017). Improved brain tumor segmentation by utilizing tumor growth model in longitudinal brain MRI. In Medical Imaging 2017: Computer-Aided Diagnosis, 10134, 101342L. https://doi.org/10.1117/12.2254034

[10] Pinto A., Pereira S., Correia H., Oliveira J., Rasteiro D. M. L. D., & Silva C. A. (2015). Brain tumour segmentation based on extremely randomized forest with high-level features. In 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 3037-3040. https://doi.org/10.1109/EMBC.2015.7319032

[11] Goodfellow I., Bengio Y., & Courville A. (2016). Deep learning. USA: MIT Press.

[12] Gole P., Bedi P., & Marwaha S. (2024). Automatic diagnosis of plant diseases via triple attention embedded vision transformer model. In International Conference on Innovative Computing and Communications: Proceedings of ICICC 2023, 2, 879-889. https://doi.org/10.1007/978-981-99-4071-4_67

[13] Gole P., Bedi P., Marwaha S., Haque A., & Deb C.K.(2023). TrIncNet: A lightweight vision transformer network for identification of plant diseases. Frontiers in Plant Science, 14, 1221557. https://doi.org/10.3389/fpls.2023.1221557

[14] Hatamizadeh A., Nath V., Tang Y., Yang D., Roth H. R., & Xu D. (2022). Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 7th International Workshop, 272-284. https://doi.org/10.1007/978-3-031-08999-2_22

[15] Wang L., Li R., Zhang C., Fang S., Duan C., Meng X., & Atkinson P. M. (2022). UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 190,196-214. https://doi.org/10.1016/J.ISPRSJPRS.2022.06.008

[16] Mihailova A., & Georgieva V. (2016). Comparative analysis various filters for noise reduction in MRI abdominal images. International Journal Information Technologies & Knowledge, 10(1), 47-66.

[17] Dehariya A. K., & Shukla P. (2021). Brain image segmentation to diagnose tumor by applying Wiener filter and intelligent water drop algorithm. International Journal of Computer Theory and Engineering, 13(3), 84-90. https://doi.org/10.7763/IJCTE.2021.V13.1294

[18] Zhang C., Shen X., Cheng H., & Qian Q. (2019). Brain tumor segmentation based on hybrid clustering and morphological operations. International Journal of Biomedical Imaging, 2019(1), 7305832. https://doi.org/10.1155/2019/7305832

[19] Arthur D. (2007). K-means++: The advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 1027-1035. https://cir.nii.ac.jp/crid/1573387451030460416

[20] Ding Y., & Fu X. (2016). Kernel-based fuzzy c-means clustering algorithm based on genetic algorithm. Neurocomputing, 188, 233-238. https://doi.org/10.1016/J.NEUCOM.2015.01.106

[21] Jayanthi S., Ranganathan H., & Palanivelan M. (2022). Segmenting brain tumour regions with fuzzy integrated active contours. IETE Journal of Research, 68(1), 514-525. https://doi.org/10.1080/03772063.2019.1615007

[22] Chan T. F., & Vese L. A. (2001). Active contours without edges. IEEE Transactions on Image Processing, 10(2), 266-277. https://doi.org/10.1109/83.902291

[23] Pereira S., Pinto A., Alves V., & Silva C. A. (2016). Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Transactions on Medical Imaging, 35(5), 1240-1251. https://doi.org/10.1109/TMI.2016.2538465

[24] Sun L., Zhang S., Chen H., & Luo L. (2019). Brain tumor segmentation and survival prediction using multimodal MRI scans with deep learning. Frontiers in Neuroscience, 13, 810. https://doi.org/10.3389/FNINS.2019.00810

[25] Wang G., Li W., Ourselin S., & Vercauteren T. (2018). Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: Third International Workshop, 178-190. https://doi.org/10.1007/978-3-319-75238-9_16

[26] Isensee F., Kickingereder P., Wick W., Bendszus M., & Maier-Hein K. H. (2018). Brain tumor segmentation and radiomics survival prediction: Contribution to the BRATS 2017 challenge. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: Third International Workshop, 287-297. https://doi.org/10.1007/978-3-319-75238-9_25

[27] Çiçek Ö., Abdulkadir A., Lienkamp S. S., Brox T., & Ronneberger O. (2016). 3D U-Net:Learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016: 19th International Conference, 424-432. https://doi.org/10.1007/978-3-319-46723-8_49

[28] Daimary D., Bora M. B., Amitab K., & Kandar D. (2020). Brain tumor segmentation from MRI images using hybrid convolutional neural networks. Procedia Computer Science, 167,2419-2428. https://doi.org/10.1016/J.PROCS.2020.03.295

[29] Balamurugan T., & Gnanamanoharan E. (2023). Brain tumor segmentation and classification using hybrid deep CNN with LuNetClassifier. Neural Computing and Applications, 35(6), 4739-4753. https://doi.org/10.1007/S00521-022-07934-7

[30] Wang W., Chen C., Ding M., Yu H., Zha S., & Li J. (2021). TransBTS: Multimodal brain tumor segmentation using transformer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 109-119. https://doi.org/10.1007/978-3-030-87193-2_11

[31] Cao H., Wang Y., Chen J., Jiang D., Zhang X., Tian Q., & Wang M. (2023). Swin-Unet: Unet-like pure transformer for medical image segmentation. In Computer Vision - ECCV 2022 Workshops,205-218. https://doi.org/10.1007/978-3-031-25066-8_9

[32] Jiang Y., Zhang Y., Lin X., Dong J., Cheng T., & Liang J.(2022). SwinBTS: A method for 3D multimodal brain tumor segmentation using Swin transformer. Brain Sciences, 12(6), 797. https://doi.org/10.3390/BRAINSCI12060797

[33] Liang J., Yang C., Zeng M., & Wang X. (2022). TransConver: Transformer and convolution parallel network for developing automatic brain tumor segmentation in MRI images. Quantit-ative Imaging in Medicine and Surgery, 12(4), 2397-2415. https://doi.org/10.21037%2Fqims-21-919

[34] Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez, A. N., :::, & Polosukhin, I. (2017). Attention is all you need. In 31st Conference on Neural Information Processing Systems,1-11.

[35] Ba J. L., Kiros J. R., & Hinton G. E. (2016). Layer normalization. arXiv Preprint: 1607.06450.

[36] Kadry S., Rajinikanth V., Raja, N. S. M., Jude Hemanth D., Hannon N. M. S., & Raj A. N. J. (2021). Evaluation of brain tumor using brain MRI with modified-moth-flame algorithm and Kapur’s thresholding: A study. Evolutionary Intelligence, 14(2), 1053-1063. https://doi.org/10.1007/S12065-020-00539-W

[37] Gagan K. R., Shlok B., & Chary V. R. (2022). MRI brain tumor segmentation using U-Net. International Journal for Research in Applied Science and Engineering Technology, 10(6), 1-7. https://doi.org/10.22214/ijraset.2022.43774

[38] Wu J., Fu R., Fang H., Zhang Y., Yang Y., Xiong,H., :::,& Xu, Y. (2024). MedSegDiff: Medical image segmentation with diffusion probabilistic model. Proceedings of Machine Learning Research, 227, 1623-1639.

[39] Sajid S., Hussain S., & Sarwar A. (2019). Brain tumor detection and segmentation in MR images using deep learning. Arabian Journal for Science and Engineering, 44(11), 9249-9261. https://doi.org/10.1007/S13369-019-03967-8