FMTNet: A Fourier-Mamba–Transformer Enhanced Network for Medical Image Segmentation

Abstract

Models based on U-shaped networks have achieved widespread success in the field of medical image segmentation, but their performance is generally limited by structural bottlenecks in the network. At this stage, feature maps experience a sharp decline in spatial resolution due to continuous downsampling, resulting in significant loss of critical boundaries and structural details. Additionally, the local receptive fields of convolutions limit the effective modelling of global context. To address this core issue, we propose a novel enhanced segmentation network called FMTNet. FMTNet fundamentally enhances the expressive power of deep features by integrating an innovative composite enhancement module at the bottleneck of the U-Net. This module consists of three synergistically working submodules: the Fourier spatial fusion module, which introduces a frequency-domain perspective to compensate for and reconstruct high-frequency structural information lost in the spatial domain; the hybrid mamba–transformer module, which efficiently captures cross-regional long-range dependencies to establish global context and the multi-scale context Aggregation module, which fuses features of different scales to adapt to objects of varying sizes. We conducted extensive experiments on multiple public multi-modal datasets, including colonoscopy polyps, dermatoscopy lesions, breast ultrasound and dental X-rays. The results demonstrate that FMTNet comprehensively outperforms SOTA methods across all key metrics, showcasing exceptional segmentation accuracy and generalisation capabilities. Our research study demonstrates that by synergistically enhancing deep features across three dimensions-frequency, global, and multi-scale-FMTNet provides a general and efficient solution to address the bottleneck issues of U-Net, significantly enhancing the accuracy and robustness of medical image segmentation. The source code and pre-trained weights are available at https://github.com/shiguiling0-has/FMTNet.

Keywords

bottleneck enhancement / Fourier transform / medical image segmentation / multi-scale feature fusion

Acknowledgements

This work was funded by UKRI (Grants EP/W020408/1 and RS718) through Doctoral Training Centre at Swansea University. Qilu Medical Talent Cultivation Project of Shandong Health Commission (Grant [2023]78). National Natural Science Foundation of China (Grant 82405459).

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

All datasets used in this study are publicly accessible.

References

[1] J. Mei, T. Zhou, K. Huang, et al., A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends,” Visual Intelligence 3, no. 1 (2025): 1, https://doi.org/10.1007/s44267-024-00071-w.

[2] W. Li, X. Nie, F. Li, Z. Huang, and G. Zeng, “FMCA-Net: A Feature Secondary Multiplexing and Dilated Convolutional Attention Polyp Segmentation Network Based on Pyramid Vision Transformer,” Expert Systems With Applications 260 (2025): 125419, https://doi.org/10.1016/j.eswa.2024.125419.

[3] J. Wu, Z. Wang, M. Hong, et al., Medical Sam Adapter: Adapting Segment Anything Model for Medical Image Segmentation,” Medical Image Analysis 102 (2025): 103547, https://doi.org/10.1016/j.media.2025.103547.

[4] S. Iqbal, T. M. Khan, S. S. Naqvi, A. Naveed, and E. Meijering, “TBConvL-Net: A Hybrid Deep Learning Architecture for Robust Medical Image Segmentation,” Pattern Recognition 158 (2025): 111028, https://doi.org/10.1016/j.patcog.2024.111028.

[5] T. Liu, S. Wang, Y. Zhao, et al., MWMF-GLRW: Using Smart Model to Accurately Predict Non-Coding RNA Interactions for Healthy Consumption,” IEEE Transactions on Consumer Electronics (2025).

[6] O. Oktay, J. Schlemper, L. L. Folgoc, et al., Attention U-Net: Learning Where to Look for the Pancreas,” preprint, arXiv:1804.03999, May 20, 2018, https://doi.org/10.48550/arxiv.1804.03999.

[7] Y. Zhao, S. Wang, Y. Zhang, et al., Dual Encoder Cross-Shape Transformer Network for Medical Image Segmentation in Internet of Medical Things for Consumer Health,” IEEE Transactions on Consumer Electronics (2025).

[8] M. A. Delsuc and P. O’connor, “The Fourier Transform in Analytical Science,” Nature Reviews Methods Primers 4, no. 1 (2024): 49, https://doi.org/10.1038/s43586-024-00326-2.

[9] Q. Zhang, D. Ram, C. Hawkins, et al., Efficient Long-Range Transformers: You Need to Attend More, But Not Necessarily at Every Layer,” preprint, arXiv:2310.12442, October 19, 2023, https://doi.org/10.48550/arXiv.2310.12442.

[10] S. Qiao, S. Pang, G. Luo, et al., DPC-MSGATNet: Dual-Path Chain Multi-Scale Gated Axial-Transformer Network for Four-Chamber View Segmentation in Fetal Echocardiography,” Complex & Intelligent Systems 9, no. 4 (2023): 4503-4519, https://doi.org/10.1007/s40747-023-00968-x.

[11] Y. Zhao, S. Wang, Y. Ren, et al., TPFIANet: Three Path Feature Progressive Interactive Attention Learning Network for Medical Image Segmentation,” Knowledge-Based Systems 323 (2025): 113778, https://doi.org/10.1016/j.knosys.2025.113778.

[12] K. O’shea and R. Nash, “An Introduction to Convolutional Neural Networks,” preprint, arXiv:1511.08458, December 2, 2015, https://doi.org/10.48550/arXiv.1511.08458.

[13] T. Liu, S. Wang, Y. Zhang, et al., Deciphering circRNA-Drug Sensitivity Associations via Global-Local Heterogeneous Matrix Factorization and Hypergraph Contrastive Learning,” Expert Systems With Applications 292 (2025): 128548, https://doi.org/10.1016/j.eswa.2025.128548.

[14] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in International Conference on Medical Image Computing and computer-assisted Intervention (Springer International Publishing, 2015), 234-241, https://doi.org/10.1007/978-3-319-24574-4_28.

[15] Y. Peng, D. Z. Chen, and M. Sonka, “U-Net v2: Rethinking the Skip Connections of u-Net for Medical Image Segmentation,” in 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI) (IEEE, 2025), 1-5.

[16] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, et al., Unet++: A Nested u-net Architecture for Medical Image Segmentation,” in International Workshop on Deep Learning in Medical Image Analysis (Springer International Publishing, 2018), 3-11.

[17] X. Xiao, S. Lian, Z. Luo, et al., Weighted Res-UNet for High-Quality Retina Vessel Segmentation,” in 2018 9th International Conference on Information Technology in Medicine and Education (ITME) (IEEE, 2018), 327-331.

[18] K. Hara, H. Kataoka, and Y. Satoh, “Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and Imagenet?,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 6546-6555.

[19] F. Milletari, N. Navab, and S. A. Ahmadi, “V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation,” in 2016 Fourth International Conference on 3D Vision (3DV) (IEEE, 2016), 565-571.

[20] L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic Image Segmentation With Deep Convolutional Nets, Atrous Convolution, and Fully Connected Crfs,” IEEE Transactions on Pattern Analysis and Machine Intelligence 40, no. 4 (2017): 834-848, https://doi.org/10.1109/tpami.2017.2699184.

[21] Y. Zhao, S. Wang, Y. Zhang, S. Qiao, and M. Zhang, “WRANet: Wavelet Integrated Residual Attention U-Net Network for Medical Image Segmentation,” Complex & Intelligent Systems 9, no. 6 (2023): 6971-6983, https://doi.org/10.1007/s40747-023-01119-y.

[22] Y. Z. Doc and S. W. Doc, “Duala-Net: A Generalizable and Adaptive Network With Dual-Branch Encoder for Medical Image Segmentation,” Computer Methods and Programs in Biomedicine 243 (2024): 107877, https://doi.org/10.1016/j.cmpb.2023.107877.

[23] A. Vaswani, N. Shazeer, N. Parmar, et al., Attention Is All You Need,” in Advances in Neural Information Processing Systems (Curran Associates Inc., 2017), 5998-6008.

[24] T. Liu, S. Wang, Y. Zhang, Y. Li, Y. Liu, and S. Huang, “TIWMFLP: Two-Tier Interactive Weighted Matrix Factorization and Label Propagation Based on Similarity Matrix Fusion for Drug-Disease Association Prediction,” Journal of Chemical Information and Modeling 64, no. 22 (2024): 8641-8654, https://doi.org/10.1021/acs.jcim.4c01589.

[25] H. Dalla-Torre, L. Gonzalez, J. Mendoza-Revilla, et al., Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics,” Nature Methods 22, no. 2 (2025): 287-297, https://doi.org/10.1038/s41592-024-02523-z.

[26] K. Han, Y. Wang, H. Chen, et al., A Survey on Vision Transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence 45, no. 1 (2022): 87-110, https://doi.org/10.1109/tpami.2022.3152247.

[27] J. M. J. Valanarasu, P. Oza, I. Hacihaliloglu, et al., Medical Transformer: Gated Axial-Attention for Medical Image Segmentation,” in International Conference on Medical Image Computing and computer-assisted Intervention (Springer International Publishing, 2021), 36-46.

[28] Z. Liu, Y. Lin, Y. Cao, et al., Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), 10012-10022, https://doi.org/10.48550/arxiv.2103.14030.

[29] P. Shamsolmoali, M. Zareapoor, E. Granger, and M. Felsberg, “Setformer Is What You Need for Vision and Language,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, no. 5, (2024), 4713-4721, https://doi.org/10.1609/aaai.v38i5.28272.

[30] J. Chen, Y. Lu, Q. Yu, et al., Transunet: Transformers Make Strong Encoders for Medical Image Segmentation,” preprint, arXiv:2102.04306, February 8, 2021, https://doi.org/10.48550/arXiv.2102.04306.

[31] H. Cao, Y. Wang, J. Chen, et al., Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation,” in European Conference on Computer Vision (Springer Nature Switzerland, 2022), 205-218, https://doi.org/10.48550/arxiv.2105.05537.

[32] W. Meng, S. Liu, and H. Wang, “AFC-Unet: Attention-Fused Full-Scale CNN-Transformer Unet for Medical Image Segmentation,” Biomedical Signal Processing and Control 99 (2025): 106839, https://doi.org/10.1016/j.bspc.2024.106839.

[33] W. Meng, A. Mu, and H. Wang, “Efficient UNet Fusion of Convolutional Neural Networks and State Space Models for Medical Image Segmentation,” Digital Signal Processing 158 (2025): 104937, https://doi.org/10.1016/j.dsp.2024.104937.

[34] S. Hao, Z. Yu, B. Zhang, et al., MEFP-Net: A Dual-Encoding Multi-Scale Edge Feature Perception Network for Skin Lesion Segmentation,” IEEE Access (2024).

[35] L. Shi, R. Zhang, Y. Wu, et al., AHC-Net: A Road Crack Segmentation Network Based on Dual Attention Mechanism and Multi-Feature Fusion,” Signal, Image and Video Processing 18, no. 6 (2024): 5311-5322, https://doi.org/10.1007/s11760-024-03234-w.

[36] P. Heckbert, “Fourier Transforms and the Fast Fourier Transform (FFT) Algorithm,” Computer Graphics 2 (1995): 15-463.

[37] A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling With Selective State Spaces,” preprint, arXiv:2312.00752, December 1, 2023, https://doi.org/10.48550/arXiv.2312.00752.

[38] J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, and F. Vilariño, “WM-DOVA Maps for Accurate Polyp Highlighting in Colonoscopy: Validation Vs. Saliency Maps From Physicians,” Computerized Medical Imaging and Graphics 43 (2015): 99-111, https://doi.org/10.1016/j.compmedimag.2015.02.007.

[39] D. Jha, P. H. Smedsrud, M. A. Riegler, et al., Kvasir-Seg: A Segmented Polyp Dataset,” in International Conference on Multimedia Modeling (Springer International Publishing, 2019), 451-462.

[40] D. Jha, P. H. Smedsrud, D. Johansen, et al., A Comprehensive Study on Colorectal Polyp Segmentation With ResUNet++, Conditional Random Field and Test-Time Augmentation,” IEEE Journal of Biomedical and Health Informatics 25, no. 6 (2021): 2029-2040, https://doi.org/10.1109/jbhi.2021.3049304.

[41] N. C. F. Codella, D. Gutman, M. E. Celebi, et al., Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC),” in IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) (IEEE, 2018), 168-172.

[42] W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy, “Dataset of Breast Ultrasound Images,” Data in Brief 28 (2020): 104863, https://doi.org/10.1016/j.dib.2019.104863.

[43] Y. Wang, Y. Zhang, X. Chen, et al., STS MICCAI 2023 Challenge: Grand Challenge on 2D and 3D Semi-Supervised Tooth Segmentation,” preprint, arXiv:2407.13246, July 18, 2024, https://doi.org/10.48550/arXiv.2407.13246.

[44] J. M. J. Valanarasu and V. M. Patel, “Unext: Mlp-Based Rapid Medical Image Segmentation Network,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer Nature Switzerland, 2022), 23-33, https://doi.org/10.48550/arxiv.2203.04967.

[45] S. Wang, X. Zhao, Y. Zhang, et al., DPMnet: Dual-Path Mlp-Based Network for Aneurysm Image Segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer Nature Switzerland, 2024), 245-254, https://doi.org/10.1007/978-3-031-72114-4_24.