A medical image segmentation model based on SAM with an integrated local multi-scale feature encoder

Abstract

Despite its remarkable performance on natural images, the segment anything model (SAM) lacks domain-specific information in medical imaging. and faces the challenge of losing local multi-scale information in the encoding phase. This paper presents a medical image segmentation model based on SAM with a local multi-scale feature encoder(LMSFE-SAM) to address the issues above. Firstly, based on the SAM, a local multi-scale feature encoder is introduced to improve the representation of features within local receptive field, thereby supplying the Vision Transformer(ViT) branch in SAM with enriched local multi-scale contextual information. At the same time, a multiaxial Hadamard product module (MHPM) is incorporated into the local multi-scale feature encoder in a lightweight manner to reduce the quadratic complexity and noise interference. Subsequently, a cross-branch balancing adapter is designed to balance the local and global information between the local multi-scale feature encoder and the ViT encoder in SAM. Finally, to obtain smaller input image size and to mitigate overlapping in patch embeddings, the size of the input image is reduced from 1 024×1 024 pixels to 256×256 pixels, and a multidimensional information adaptation component is developed, which includes feature adapters, position adapters, and channel-spatial adapters. This component effectively integrates the information from small-sized medical images into SAM, enhancing its suitability for clinical deployment. The proposed model demonstrates an average enhancement ranging from 0.038 7 to 0.319 1 across six objective evaluation metrics on BUSI, DDTI, and TN3K datasets compared to eight other representative image segmentation models. This significantly enhances the performance of the SAM on medical images, providing clinicians with a powerful tool in clinical diagnosis.

Keywords

segment anything model (SAM) / medical image segmentation / encoder / decoder / multiaxial Hadamard product module (MHPM) / cross-branch balancing adapter

References

[1] LITJENS G, KOOI T, BEJNORDI B E, et al. A survey on deep learning in medical image analysis. Medical Image Analysis, 2017, 42: 60-88.

[2] ZHOU S K, GREENSPAN H, DAVATZIKOS C, et al. A review of deep learning in medical imaging: imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE, 2021, 109(5): 820-838.

[3] WU S Y, ZHUO G P, ZHU J D, et al. Segmentation and classification algorithm of retinal arteriovenous vessel based on improved U-Net. Journal of North University of China(Natural Science Edition), 2023, 44(1): 79-85.

[4] CHAI R, LUO Y B, QIN P L, et al.Colon cancer gland segmentation network based on edge fusion and multi scale feature enhancement. Journal of North University of China(Natural Science Edition), 2025, 46(4): 411-421.

[5] FABIAN I, PAUL F J, KOHL S A A, et al. NNU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 2021, 18(2): 203-211.

[6] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation//2015 IEEE Conference on Computer Vision and Pattern Recognition, June 7-12, 2015, Boston, MA, USA. New York: IEEE, 2015: 3431-3440.

[7] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. October 5-9, 2015, Munich, Germany. Cham: Springer International Publishing, 2015: 234-241.

[8] LI C X, LIU X Y, LI W Y, et al. U-KAN makes strong backbone for medical image segmentation and generation. 2024: 2406.02918.

[9] WU H S, CHEN S H, CHEN G L, et al. FAT-Net: Feature adaptive transformers for automated skin lesion segmentation. Medical Image Analysis, 2022, 76: 102327.

[10] WANG H, CAO P, WANG J, et al. UCTransNet: rethinking the skip connections in U-net from a channel-wise perSpective with transformer//AAAI Conference on Artificial Intelligence, February 22-March 1, 2022, Vancouver, Canada. New York: AAAI Press, 2022, 36(3): 2441-2449.

[11] YUAN F, ZHANG Z X, FANG Z J. An effective CNN and transformer complementary network for medical image segmentation. Pattern Recognition, 2023, 136: 109228.

[12] LI G, CHEN R Y, ZHANG J, et al. Fusing enhanced transformer and large kernel CNN for malignant thyroid nodule segmentation. Biomedical Signal Processing and Control, 2023, 83: 104636.

[13] PANG C, LU X Q, LIU X, et al. IIAM: intra and inter attention with mutual consistency learning network for medical image segmentation. IEEE Journal of Biomedical and Health Informatics, 2024, 28(10): 5971-5983.

[14] KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything//2023 IEEE/CVF International Conference on Computer Vision, October 1-6, 2023, Paris, France. New York: IEEE, 2023: 3992-4003.

[15] DOSOVITSKIY A. An image is worth 16x16 words: transformers for image recognition at scale. (2010-10-22)[2024-08-26].

[16] HE S, BAO R, LI J, et al. Computer-vision benchmark segment-anything model (SAM) in medical images: accuracy in 12 datasets. (2023-04-18) [2024-09-01].

[17] XIONG Y Y, VARADARAJAN B, WU L M, et al. EfficientSAM: leveraged masked image pretraining for efficient segment anything//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 16-22, 2024, Seattle, WA, USA. New York: IEEE, 2024: 16111-16121.

[18] ZHANG Y C, SHEN Z R, JIAO R S. Segment anything model for medical image segmentation: current applications and future directions. Computers in Biology and Medicine, 2024,171: 108238.

[19] HUANG Y H, YANG X, LIU L, et al. Segment anything model for medical images? Medical Image Analysis, 2024, 92: 103061.

[20] MA J, HE Y T, LI F F, et al. Segment anything in medical images. Nature Communications,2024,15(1): 654.

[21] CHENG J, YE J, DENG Z, et al. SAM-MED2D. (2023-08-30) [2024-04-10].

[22] HU M Z, LI Y H, YANG X F. BreastSAM: adapting the segmentation anything model for breast tumor segmentation in ultrasound imaging//Medical Imaging 2024: Ultrasonic Imaging and Tomography, February 18-22, 2024, San Diego, California, USA. Bellingham: SPIE, 2024, 12932: 182-196.

[23] AL-DHABYANI W, GOMAA M, KHALED H, et al. Dataset of breast ultrasound images. Data in Brief, 2020, 28: 104863.

[24] PEDRAZA L, VARGAS C, NARVÁEZ F, et al. An open access thyroid ultrasound image database//10th International Symposium on Medical Information Processing and Analysis, October 14-16, 2015, Cartagena de Indias, Colombia, USA. Bellingham: SPIE, 2015, 9287: 188-193.

[25] GONG H F, CHEN J X, CHEN G Q, et al. Thyroid region prior guided attention for ultrasound segmentation of thyroid nodules. Computers in Biology and Medicine, 2023, 155: 106389.

[26] ZHOU C, LI X T, LOY C C, et al. EdgeSAM: prompt-In-the-loop distillation for on-device deployment of SAM. 2023: 2312.06660.

[27] REZA S, AMIN O B, HASHEM M M A. TransResUNet: improving U-Net architecture for robust lungs segmentation in chest X-rays//2020 IEEE Region 10 Symposium, June 5-7, 2020, Dhaka, Bangladesh. New York: IEEE, 2020: 1592-1595.