MOSformer: Momentum encoder-based inter-slice fusion transformer for medical image segmentation

Abstract

Medical image segmentation takes an important position in various clinical applications. 2.5D-based segmentation models bridge the computational efficiency of 2D-based models with the spatial perception capabilities of 3D-based models. However, existing 2.5D-based models primarily adopt a single encoder to extract features of target and neighborhood slices, failing to effectively fuse inter-slice information, resulting in suboptimal segmentation performance. In this study, a novel momentum encoder-based inter-slice fusion transformer (MOSformer) is proposed to overcome this issue by leveraging inter-slice information from multi-scale feature maps extracted by different encoders. Specifically, dual encoders are employed to enhance feature distinguishability among different slices. One of the encoders is moving-averaged to maintain consistent slice representations. Moreover, an inter-slice fusion transformer (IF-Trans) module is developed to fuse inter-slice multi-scale features. MOSformer is evaluated on three benchmark datasets (Synapse, ACDC, and AMOS), achieving a new state-of-the-art with 85.63%, 92.19%, and 85.43% DSC, respectively. These results demonstrate MOSformer’s competitiveness in medical image segmentation.

Keywords

Medical image segmentation / Momentum encoder / Inter-slice fusion / Transformer

CRediT authorship contribution statement

Dexing Huang: Writing – original draft, Methodology, Data curation. Xiaohu Zhou: Funding acquisition, Conceptualization. Meijiang Gui: Resources, Funding acquisition. Xiaoliang Xie: Funding acquisition. Shiqi Liu: Resources. Shuangyi Wang: Resources. Zhenqiu Feng: Resources. Zhichao Lai: Resources. Zengguang Hou: Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China (2023YFC2415100), in part by the National Natural Science Foundation of China (62222316, 62373351, 82327801, 62073325, 62303463), in part by the Chinese Academy of Sciences Project for Young Scientists in Basic Research (YSBR-104), in part by the Beijing Natural Science Foundation (F252068, 4254107), in part by Beijing Nova Program, China (20250484813), in part by China Postdoctoral Science Foundation, China (2024M763535), in part by the Postdoctoral Fellowship Program of CPSF (GZC20251170) and in part by CAMS Innovation Fund for Medical Sciences (ClFMS) (2023-I2M-C&T-B-017).

References

[1] Y.-J. Zhou, X.-L. Xie, X.-H. Zhou, S.-Q. Liu, G.-B. Bian, Z.-G. Hou, A real-time multifunctional framework for guidewire morphological and positional analysis in interventional X-ray fluoroscopy, IEEE Trans. Cogn. Dev. Syst. 13 (3) (2020) 657-667.

[2] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B.A. Landman, G. Litjens, B. Menze, O. Ronneberger, R.M. Summers, et al., The medical segmentation decathlon, Nat. Commun. 13 (1) (2022) 4128.

[3] R.-Q. Li, et al., A unified framework for multi-guidewire endpoint localization in fluoroscopy images, IEEE Trans. Biomed. Eng. 69 (4) (2021) 1406-1416.

[4] R.-Q. Li, et al., Real-time multi-guidewire endpoint localization in fluoroscopy images, IEEE Trans. Med. Imaging 40 (8) (2021) 2002-2014.

[5] Z. Han, H. Tian, X. Han, J. Wu, W. Zhang, C. Li, L. Qiu, X. Duan, W. Tian, A respiratory motion prediction method based on LSTM-AE with attention mechanism for spine surgery, Cyborg Bionic Syst. 5 (2024) 0063.

[6] D.-X. Huang, X.-H. Zhou, X.-L. Xie, S.-Q. Liu, Z.-Q. Feng, Z.-G. Hou, N. Ma, L. Yan, Real-time 2D/3D registration via CNN regression and centroid alignment, IEEE Trans. Autom. Sci. Eng. 22 (2025) 85-98.

[7] Y. Chen, G. Li, C. Li, W. Yu, Z. Fan, J. Bai, S. Tu, GVM-Net: A GNN-based vessel matching network for 2D/3D non-rigid coronary artery registration, IEEE Trans. Med. Imaging, (2025), https://doi.org/10.1109/TMI.2025.3540906.

[8] J. Zhang, L. Liu, P. Xiang, Q. Fang, X. Nie, H. Ma, J. Hu, R. Xiong, Y. Wang, H. Lu, AI co-pilot bronchoscope robot, Nat. Commun. 15 (1) (2024) 241.

[9] L. Li, X. Li, B. Ouyang, H. Mo, H. Ren, S. Yang, Three-dimensional collision avoidance method for robot-assisted minimally invasive surgery, Cyborg Bionic Syst. 4 (2023) 0042.

[10] X.-H. Zhou, X.-L. Xie, S.-Q. Liu, Z.-L. Ni, Y.-J. Zhou, R.-Q. Li, M.-J. Gui, C.-C. Fan, Z.-Q. Feng, G.-B. Bian, et al., Learning skill characteristics from manipulations, IEEE Trans. Neural Netw. Learn. Syst. 34 (12) (2023) 9727-9741.

[11] L.-S. Zhang, et al., Novel 3D instrument navigation in intracranial vascular surgery with multi-source image fusion and self-calibration, Biomim. Intell. Robot. (2025), 100233.

[12] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional networks for biomedical image segmentation, in: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions, MICCAI, 2015, pp. 234-241.

[13] Z. Zhou, M.M.R. Siddiquee, N. Tajbakhsh, J. Liang, UNet++: Redesigning skip connections to exploit multiscale features in image segmentation, IEEE Trans. Med. Imaging 39 (6) (2019) 1856-1867.

[14] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, J. Wu, UNet 3+: A full-scale connected UNet for medical image segmentation, in: Proceedings of Th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2020, pp. 1055-1059.

[15] Z.-L. Ni, et al., SurgiNet: Pyramid attention aggregation and class-wise self-distillation for surgical instrument segmentation, Med. Image Anal. 76 (2022) 102310.

[16] D.-X. Huang, X.-H. Zhou, X.-L. Xie, S.-Q. Liu, S.-Y. Wang, Z.-Q. Feng, M.-J. Gui, H. Li, T.-Y. Xiang, B.-X. Yao, et al., SPIRONet: Spatial-frequency learning and topological channel interaction network for vessel segmentation, (2024),arXiv:2406.19749.

[17] N. Tajbakhsh, L. Jeyaseelan, Q. Li, J.N. Chiang, Z. Wu, X. Ding, Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation, Med. Image Anal. 63 (2020) 101693.

[18] Y. Zhang, Q. Liao, L. Ding, J. Zhang, Bridging 2D and 3D segmentation networks for computation-efficient volumetric medical image segmentation: An empirical study of 2.5d solutions, Comput. Med. Imaging Graph. (2022), 102088.

[19] R. Azad, E.K. Aghdam, A. Rauland, Y. Jia, A.H. Avval, A. Bozorgpour, S. Karimijafarbigloo, J.P. Cohen, E. Adeli, D. Merhof, Medical image segmentation review: The success of U-Net, IEEE Trans. Pattern Anal. Mach. Intell. 46 (12) (2024) 10076-10095.

[20] L. Mou, Y. Zhao, H. Fu, Y. Liu, J. Cheng, Y. Zheng, P. Su, J. Yang, L. Chen, A.F. Frangi, et al., CS²-net: Deep learning segmentation of curvilinear structures in medical imaging, Med. Image Anal. 67 (2021) 101874.

[21] K. Roy, D. Banik, D. Bhattacharjee, O. Krejcar, C. Kollmann, LwMLA-NET: A lightweight multi-level attention-based network for segmentation of COVID-19 lungs abnormalities from CT images, IEEE Trans. Instrum. Meas. 71 (2022) 5007813.

[22] H. Cao, Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, M. Wang, Swin-Unet: Unet-like pure transformer for medical image segmentation, in: Proceedings of the European Conference on Computer Vision Workshops, ECCVW, 2022, pp. 205-218.

[23] J. Chen, J. Mei, X. Li, Y. Lu, Q. Yu, Q. Wei, X. Luo, Y. Xie, E. Adeli, Y. Wang, et al., TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers, Med. Image Anal. 97 (2024) 103280.

[24] Ö. Çiçek, A. Abdulkadir, S.S. Lienkamp, T. Brox, O. Ronneberger, 3D U-Net: learning dense volumetric segmentation from sparse annotation, in: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions, MICCAI, 2016, pp. 424-432.

[25] L. Yu, J.-Z. Cheng, Q. Dou, X. Yang, H. Chen, J. Qin, P.-A. Heng, Automatic 3D cardiovascular MR segmentation with densely-connected volumetric convnets, in: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions, MICCAI, 2017, pp. 287-295.

[26] J. Chen, L. Yang, Y. Zhang, M. Alber, D.Z. Chen, Combining fully convolutional and recurrent neural networks for 3D biomedical image segmentation, Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 29, (2016).

[27] X. Yan, H. Tang, S. Sun, H. Ma, D. Kong, X. Xie, AFTer-UNet: Axial fusion transformer UNet for medical image segmentation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2022, pp. 3971-3981.

[28] A.L.Y. Hung, H. Zheng, Q. Miao, S.S. Raman, D. Terzopoulos, K. Sung, CAT-Net: A cross-slice attention transformer model for prostate zonal segmentation in MRI, IEEE Trans. Med. Imaging 42 (1) (2022) 291-303.

[29] Y. Zhang, L. Yuan, Y. Wang, J. Zhang, SAU-Net: Efficient 3D spine MRI segmentation using inter-slice attention, in: Proceedings of the Medical Imaging with Deep Learning, MIDL, 2020, pp. 903-913.

[30] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin Transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021, pp. 10012-10022.

[31] Q. Yu, L. Xie, Y. Wang, Y. Zhou, E.K. Fishman, A.L. Yuille, Recurrent saliency transformation network: Incorporating multi-stage visual cues for small organ segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2018, pp. 8280-8289.

[32] Y. Li, H. Li, Y. Fan, ACEnet: Anatomical context-encoding network for neuroanatomy segmentation, Med. Image Anal. 70 (2021) 101991.

[33] X. Yang, L. Yu, S. Li, H. Wen, D. Luo, C. Bian, J. Qin, D. Ni, P.-A. Heng, Towards automated semantic segmentation in prenatal volumetric ultrasound, IEEE Trans. Med. Imaging 38 (1) (2018) 180-193.

[34] B. Pang, K. Zha, H. Cao, C. Shi, C. Lu, Deep RNN framework for visual sequential applications, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2019, pp. 423-432.

[35] L. Li, S. Lian, Z. Luo, S. Li, B. Wang, S. Li, Learning consistency-and discrepancy-context for 2D organ segmentation, in: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions, MICCAI, 2021, pp. 261-270.

[36] D. Guo, D. Terzopoulos, A transformer-based network for anisotropic 3D medical image segmentation, in: Proceedings of the IEEE International Conference on Image Processing, ICIP, 2021, pp. 8857-8861.

[37] A.L.Y. Hung, H. Zheng, K. Zhao, X. Du, K. Pang, Q. Miao, S.S. Raman, D. Terzopoulos, K. Sung, CSAM: A 2.5 D cross-slice attention module for anisotropic volumetric medical image segmentation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2024, pp. 5923-5932.

[38] A. Kumar, et al., A flexible 2.5 d medical image segmentation approach with in-slice and cross-slice attention, Comput. Biol. Med. 182 (2024) 109173.

[39] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, in: Proceedings of the International Conference on Learning Representations, ICLR, 2020.

[40] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: Proceedings of the European Conference on Computer Vision, ECCV, 2020, pp. 213-229.

[41] P. Lijin, M. Ullah, A. Vats, F.A. Cheikh, M.S. Nair, et al., Dual encoder decoder shifted window-based transformer network for polyp segmentation with self-learning approach, IEEE Trans. Artif. Intell. 5 (7) (2024) 3456-3469.

[42] Y. Chenyu, et al., Class-aware generative adversarial transformers for medical image segmentation, in: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022, pp. 29582-29596.

[43] X. Huang, Z. Deng, D. Li, X. Yuan, Y. Fu, MISSFormer: An effective transformer for 2D medical image segmentation, IEEE Trans. Med. Imaging 42 (5) (2023) 1484-1494.

[44] A. Hatamizadeh, et al., UNETR: Transformers for 3D medical image segmentation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2022, pp. 574-584.

[45] Y. Xie, J. Zhang, C. Shen, Y. Xia, CoTr: Efficiently bridging cnn and transformer for 3D medical image segmentation, in: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions, MICCAI, 2021, pp. 171-180.

[46] H.-Y. Zhou, et al., nnFormer: Volumetric medical image segmentation via a 3D transformer, IEEE Trans. Image Process. 32 (2023) 4036-4045.

[47] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2016, pp. 770-778.

[48] K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2020, pp. 9729-9738.

[49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 30, (2017).

[50] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, Z. Tu, Deeply-supervised nets, in: Proceedings of the International Conference on Artificial Intelligence and Statistics, AISTATS, 2015, pp. 562-570.

[51] B. Landman, Z. Xu, J. Igelsias, M. Styner, T. Langerak, A. Klein, MICCAI multi-atlas labeling beyond the cranial vault–workshop and challenge, Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions Workshops, vol. 5, MICCAIW, (2015), p. 12.

[52] O. Bernard, et al., Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved?, IEEE Trans. Med. Imaging 37 (11) (2018) 2514-2525.

[53] Y. Ji, H. Bai, C. Ge, J. Yang, Y. Zhu, R. Zhang, Z. Li, L. Zhanng, W. Ma, X. Wan, et al., AMOS: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation, Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 36722-36732.

[54] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, D. Rueckert, Attention gated networks: Learning to leverage salient regions in medical images, Med. Image Anal. 53 (2019) 197-207.

[55] H. Wang, S. Xie, L. Lin, Y. Iwamoto, X.-H. Han, Y.-W. Chen, R. Tong, Mixed transformer U-Net for medical image segmentation, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2022, pp. 2390-2394.

[56] H. Wang, P. Cao, J. Wang, O.R. Zaiane, UCTransNet: Rethinking the skip connections in U-Net from a channel-wise perspective with transformer, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, AAAI, 2022, pp. 2441-2449.

[57] M. Heidari, A. Kazerouni, M. Soltany, R. Azad, E.K. Aghdam, J. Cohen-Adad, D. Merhof, HiFormer: Hierarchical multi-scale representations using transformers for medical image segmentation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, WACV, 2023, pp. 6202-6212.

[58] K. Zhang, D. Liu, Customized segment anything model for medical image segmentation, (2023),arXiv:2304.13785.

[59] F. Milletari, N. Navab, S.-A. Ahmadi, V-Net: Fully convolutional neural networks for volumetric medical image segmentation, in: Proceedings of the International Conference on 3D Vision (3DV), 2016, pp. 565-571.

[60] A. Hatamizadeh, V. Nath, Y. Tang, D. Yang, H.R. Roth, D. Xu, Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images, in: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Interventions Workshops, MICCAIW, 2021, pp. 272-284.

[61] N.-T. Bui, et al., SAM3D: Segment anything model in volumetric medical images, Proceedings of the IEEE International Symposium on Biomedical Imaging, ISBI, (2024), https://doi.org/10.1109/ISBI56570.2024.10635844.

[62] W. Zhang, Y. Zhang, L. Zhang, Multi-planar data augmentation and lightweight skip connection design for deep learning based abdominal CT image segmentation, IEEE Trans. Instrum. Meas. 72 (2023) 2532111.

[63] L. Van der Maaten, G. Hinton, Visualizing data using t-SNE., J. Mach. Learn. Res., 9 (11), (2008).

[64] R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2017, pp. 618-626.