Image segmentation network for laparoscopic surgery

Abstract

Surgical image segmentation serves as the foundation for laparoscopic surgical navigation technology. The indistinct local features of biological tissues in laparoscopic image pose challenges for image segmentation. To address this issue, we develop an image segmentation network tailored for laparoscopic surgery. Firstly, we introduce the Mixed Attention Enhancement (MAE) module that sequentially conducts the Channel Attention Enhancement (CAE) module and the Global Feature Enhancement (GFE) module linked in series. The CAE module enhances the network’s perception of prominent channels, allowing feature maps to exhibit clear local features. The GFE module is capable of extracting global features from both the height and width dimensions of images and integrating them into three-dimensional features. This enhancement improves the network’s ability to capture global features, thereby facilitating the inference of regions with indistinct local features. Secondly, we propose the Multi-scale Feature Fusion (MFF) module. This module expands the feature map into various scales, further enlarging the network’s receptive field and enhancing perception of features at multiple scales. In addition, we tested the proposed network on the EndoVis 2018 and a human minimally invasive liver resection image segmentation dataset, comparing it against six other advanced image segmentation networks. The comparative test results demonstrate that the proposed network achieves the most advanced performance on both datasets, proving its potential in improving surgical image segmentation outcome. The codes of MAMNet are available at: https://github.com/Pang1234567/MAMNet.

Keywords

Laparoscopic surgery image / Medical image segmentation / Convolutional neural networks / Attention mechanism / Feature fusion

CRediT authorship contribution statement

Kang Peng: Writing - review & editing, Writing - original draft, Visualization, Software, Methodology, Data curation, Conceptualization. Yaoyuan Chang: Writing - review & editing, Writing - original draft, Software, Methodology, Formal analysis. Guodong Lang: Writing - review & editing, Validation, Software, Formal analysis. Jian Xu: Writing - review & editing, Validation, Formal analysis, Data curation. Yongsheng Gao: Validation, Supervision, Resources, Project administration, Funding acquisition. Jiajun Yin: Validation, Supervision, Resources, Project administration, Data curation. Jie Zhao: Supervision, Resources, Project administration, Data curation.

Ethics approval

This study was conducted in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Ethical approval for the use of the minimally invasive liver resection dataset was obtained from the Ethics Committee of Zhongshan Hospital Affiliated to Dalian University (Approval No. KY2023-002-2, Date: April 3, 2024). Written informed consent was obtained from all individual participants included in the study.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by National Key Research and De-velopment Program of China (2022YFB4700700), the Dalian Deng Feng Program: key medical specialties in construction funded by the People’s Government of Dalian Municipality, China ([2021]243), and the Liaoning Provincial Natural Science Foundation, China (2023JH2/101300102).

References

[1] Nazim Haouchine, Jeremie Dequidt, Igor Peterlik, Erwan Kerrien, Marie-Odile Berger, Stéphane Cotin, Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery, in: 2013 IEEE International Symposium on Mixed and Augmented Reality, ISMAR, IEEE, 2013, pp. 199-208.

[2] Toby Collins, Daniel Pizarro, Simone Gasparini, Nicolas Bourdel, Pauline Chauvet, Michel Canis, Lilian Calvet, Adrien Bartoli,Augmented reality guided laparoscopic surgery of the uterus, IEEE Trans. Med.Imaging 40 (1) (2020) 371-380.

[3] Duygu Sarikaya, Jason J. Corso, Khurshid A. Guru, Detection and localiza-tion of robotic tools in robot-assisted surgery videos using deep neural networks for region proposal and detection, IEEE Trans. Med. Imaging 36(7) (2017) 1542-1549.

[4] Zhen-Liang Ni, Gui-Bin Bian, Zhen Li, Xiao-Hu Zhou, Rui-Qi Li, Zeng-Guang Hou, Space squeeze reasoning and low-rank bilinear feature fusion for surgical image segmentation, IEEE J. Biomed. Heal. Inform. 26 (7) (2022) 3209-3217.

[5] Zhen Liang Ni,Pyramid attention aggregation network for semantic seg-mentation of surgical instruments, in:The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI-20, 2020.

[6] Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kad-khodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al., 2018 robotic scene segmentation challenge, 2020, arXiv preprint arXiv:2001.11190.

[7] Zhen-Liang Ni, Gui-Bin Bian, Xiao-Hu Zhou, Zeng-Guang Hou, Xiao-Liang Xie, Chen Wang, Yan-Jie Zhou, Rui-Qi Li, Zhen Li, Raunet: Residual attention u-net for semantic segmentation of cataract surgical instruments,in:, International Conference on Neural Information Processing, Springer, 2019, pp. 139-149.

[8] Wenting Shen, Yaonan Wang, Min Liu, Jiazheng Wang, Renjie Ding, Zhe Zhang, Erik Meijering,Branch aggregation attention network for robotic surgical instrument segmentation, IEEE Trans. Med. Imaging (2023).

[9] Ahmed Mohammed, Sule Yildirim, Ivar Farup, Marius Pedersen, Øistein Hovde, Streoscennet: surgical stereo robotic scene segmentation, in: Med-ical Imaging 2019: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 10951, SPIE, 2019, pp. 174-182.

[10] Eung-Joo Lee, William Plishker, Xinyang Liu, Timothy Kane, Shuvra S Bhattacharyya, Raj Shekhar, Segmentation of surgical instruments in la-paroscopic videos: training dataset generation and deep-learning-based framework, in: Medical Imaging 2019: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 10951, SPIE, 2019, pp. 461-469.

[11] S.M. Kamrul Hasan, Cristian A. Linte, U-NetPlus: A modified encoder-decoder U-Net architecture for semantic and instance segmentation of surgical instruments from laparoscopic images, in: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC, IEEE, 2019, pp. 7205-7211.

[12] Yanwen Sun, Bo Pan, Yili Fu, Lightweight deep neural network for real-time instrument semantic segmentation in robot assisted minimally invasive surgery, IEEE Robot. Autom. Lett. 6 (2) (2021) 3870-3877.

[13] Lei Yang, Yuge Gu, Guibin Bian, Yanhong Liu, DRR-Net: A dense-connected residual recurrent convolutional network for surgical instrument segmen-tation from endoscopic images, IEEE Trans. Med . Robot. Bionics 4 (3) (2022) 696-707.

[14] Lei Yang, Yuge Gu, Guibin Bian, Yanhong Liu, TMF-Net: A transformer-based multiscale fusion network for surgical instrument segmentation from endoscopic images, IEEE Trans. Instrum. Meas. 72 (2022) 1-15.

[15] Laurent Itti, Christof Koch, Ernst Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Mach. Intell. 20 (11) (1998) 1254-1259.

[16] Xiaoling Yang, Xinjian Chen, Dehui Xiang, Attention-guided channel to pixel convolution network for retinal layer segmentation with choroidal neovascularization, in: Medical Imaging 2020: Image Processing, vol. 11313, SPIE, 2020, pp. 786-792.

[17] Ran Gu, Guotai Wang, Tao Song, Rui Huang, Michael Aertsen, Jan Deprest, Sébastien Ourselin, Tom Vercauteren, Shaoting Zhang,CA-Net: Compre-hensive attention convolutional neural networks for explainable medical image segmentation, IEEE Trans. Med. Imaging 40 (2) (2020) 699-711.

[18] Kumar Rajamani, Sahana D. Gowda, Vishwa Nedunoori Tej, Sriv-idya Tirunellai Rajamani, Deformable attention (DANet) for semantic image segmentation, in: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society, EMBC, IEEE, 2022, pp. 3781-3784.

[19] Lei Yang, Yuge Gu, Guibin Bian, Yanhong Liu, An attention-guided network for surgical instrument segmentation from endoscopic images, Comput. Biol. Med. 151 (2022) 106216.

[20] Yiming Wang, Yan Hu, Junyong Shen, Xiaoqing Zhang, Heng Li, Zhongxi Qiu, Fangfu Ye, Jiang Liu, CGBA-Net: context-guided bidirectional attention network for surgical instrument segmentation, Int. J. Comput. Assist. Radiol. Surg. 18 (10) (2023) 1769-1781.

[21] Lei Yang, Hongyong Wang, Guibin Bian, Yanhong Liu,HCTA-Net: A hybrid CNN-transformer attention network for surgical instrument segmentation, IEEE Trans. Med. Robot. Bionics (2023).

[22] Shokofeh Anari, Gabriel Gomes De Oliveira, Ramin Ranjbarzadeh, An-gela Maria Alves, Gabriel Caumo Vaz, Malika Bendechache, Efficien-tUNetViT: Efficient breast tumor segmentation utilizing UNet architecture and pretrained vision transformer, Bioengineering (Basel) 11 (9) (2024).

[23] Zixuan Wang, Ruofan Wu, Yanran Xu, Yi Liu, Ruimei Chai, He Ma, A two-stage CNN method for MRI image segmentation of prostate with lesion, Biomed. Signal Process. Control. 82 (2023) 104610.

[24] Chao Zhuang, Tianyi Ma, Bokai Xuan, Cheng Chang, Baichuan An, Minghuan Yin, Hao Sun, Deep learning-based semantic segmentation of human features in bath scrubbing robots, Biomim. Intell. Robot. 4 (1) (2024).

[25] Peipei Li, Zhao Qiu, Yuefu Zhan, Huajing Chen, Sheng Yuan, Multi-scale bottleneck residual network for retinal vessel segmentation, J. Med. Syst. 47 (1) (2023) 102.

[26] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia,Pyramid scene parsing network, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2881-2890.

[27] Ruicong Zhang, Li Zhuo, Hui Zhang, Yan Zhang, Jinman Kim, Hongxia Yin, Pengfei Zhao, Zhenchang Wang, Vestibule segmentation from CT images with integration of multiple deep feature fusion strategies, Comput. Med. Imaging Graph. 89 (2021) 101872.

[28] Zongyuan Ding, Tao Wang, Quansen Sun, Hongyuan Wang, Adaptive fusion with multi-scale features for interactive image segmentation, Appl. Intell. 51 (8) (2021) 5610-5621.

[29] Ramin Ranjbarzadeh, Payam Zarbakhsh, Annalina Caputo, Erfan Babaee Tirkolaee, Malika Bendechache, Brain tumor segmentation based on op-timized convolutional neural network and improved chimp optimization algorithm, Comput. Biol. Med. 168, 107723.

[30] Dawei Yang, Yan Du, Hongli Yao, Liyan Bao, Image semantic segmentation with hierarchical feature fusion based on deep neural network, Connect. Sci. 34 (1) (2022) 1772-1784.

[31] ZhengLi Zhai, Shu Feng, Luyao Yao, Penghui Li, Retinal vessel image segmentation algorithm based on encoder-decoder structure, Multimedia Tools Appl. 81 (23) (2022) 33361-33373.

[32] Zhenzhong Liu, Laiwang Zheng, Shubin Yang, Zichen Zhong, Guobin Zhang, MFF-Net: Multiscale feature fusion semantic segmentation network for intracranial surgical instruments, Int. J. Med. Robot. Comput. Assist. Surg. 20 (1) (2024) e2595.

[33] Tahir Mahmood, Jin Seong Hong, Nadeem Ullah, Sung Jae Lee, Abdul Wahid, Kang Ryoung Park, CFFR-Net: A channel-wise features fusion and recalibration network for surgical instruments segmentation, Eng. Appl. Artif. Intell. 126 (2023) 107096.

[34] Jinlian Du, Yanqiu Zhang, Xueyun Jin, Xiao Zhang, A cell image segmen-tation method based on edge feature residual fusion, Methods 219 (2023) 111-118.

[35] LiFang Chen, Jiawei Li, Hongze Ge, TBUnet: A pure convolutional U-Net capable of multifaceted feature extraction for medical image segmentation, J. Med. Syst. 47 (1) (2023) 122.

[36] Yunfeng Wang, Yi Zhou, Hao Wu, Xiyu Liu, Xiaodi Zhai, Kuizhi Sun, Chengliang Tian, Haixia Zhao, Tao Li, Wenguang Jia, et al., MFCANet: A road scene segmentation network based on multi-scale feature fusion and context information aggregation, J. Vis. Commun. Image Represent. 98 (2024) 104055.

[37] Xun Wang, Xudong Zhang, Gan Wang, Ying Zhang, Xin Shi, Huanhuan Dai, Min Liu, Zixuan Wang, Xiangyu Meng, TransFusionNet: Semantic and spatial features fusion framework for liver tumor and vessel segmentation under JetsonTX2, IEEE J. Biomed. Heal. Inform. 27 (3) (2022) 1173-1184.

[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun,Deep residual learning for image recognition, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.

[39] Jonathan Long, Evan Shelhamer, Trevor Darrell,Fully convolutional net-works for semantic segmentation, in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431-3440.

[40] Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, Garrison Cottrell, Understanding convolution for semantic segmentation, in: 2018 IEEE Winter Conference on Applications of Computer Vision, WACV, Ieee, 2018, pp. 1451-1460.