Semi-Supervised Instrument Segmentation for Endoscopic Spinal Surgery

Abstract

Segmentation tasks require multiple annotation work which is time-consuming and labour-intensive. How to make full use of unlabelled data to assist in training deep learning models has been a research hotspot in recent years. This paper takes in-strument segmentation in endoscopic surgery as the background to explore how to use unlabelled data for semi-supervised learning more reasonably and effectively. An adaptive gradient correction method based on the degree of perturbation is proposed to improve segmentation accuracy. This paper integrates the recently popular segment anything model (SAM) with semi-supervised learning, taking full advantage of the large model to enhance the zero-shot ability of the model. Experimental results demonstrate the superior performance of the proposed segmentation strategy compared to traditional semi-supervised segmentation methods, achieving a 2.56% improvement in mean intersection over union (mIoU). The visual segmentation results show that incorporation of SAM significantly enhances our method, resulting in more accurate segmentation boundaries.

Keywords

deep learning / image segmentation / intelligent robots / robotics

References

[1] M. Kim, H. S. Kim, S. W. Oh, et al. “Evolution of Spinal Endoscopic Surgery,” , Neurospine 16, no. 1 (2019): 6-14, https://doi.org/10.14245/ns.1836322.161.

[2] L. Qiu, C. Li, and H. Ren, “Real-Time Surgical Instrument Tracking in Robot-Assisted Surgery Using Multi-Domain Convolutional Neural Network,” Healthcare Technology Letters 6, no. 6 (2019): 159-164, https://doi.org/10.1049/htl.2019.0068.

[3] C. Li, X. Li, K. Wang, W. Chen, Q. Liu, and X. Duan, “Self-Supervised Monocular Depth Estimation for Endoscopic Imaging,” IEEE Journal of Biomedical and Health Informatics (2024): 1-11, https://doi.org/10.1109/jbhi.2024.3434372.

[4] D. Xie, W. Chen, J. Zhao, et al., “Surgical Instruments Hyalinization: Occlusion Removal in Minimally Invasive Endoscopic Surgery,” Bio-mimetic Intelligence and Robotics 3, no. 3 (2023): 100105, https://doi.org/10.1016/j.birob.2023.100105.

[5] Y. Jin, K. Cheng, Q. Dou, and P. A. Heng, “Incorporating Temporal Prior From Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, 2019), 440-448.

[6] C. González, L. Bravo-Sánchez, and P. Arbelaez, “Isinet: An Instance-Based Approach for Surgical Instrument Segmentation,” in Interna-tional Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, 2020), 595-605.

[7] L. Yang, H. Wang, Y. Gu, G. Bian, Y. Liu, and H. Yu, “TMA-Net: A Transformer-Based Multi-Scale Attention Network for Surgical Instru-ment Segmentation,” IEEE Transactions on Medical Robotics and Bionics 5, no. 2 (2023): 323-334, https://doi.org/10.1109/tmrb.2023.3269856.

[8] S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image Segmentation Using Deep Learning: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence 44, no. 7 (2021): 3523-3542.

[9] X. Chen, Y. Yuan,G. Zeng, and J. Wang, “Semi-supervised Semantic Segmentation With Cross Pseudo Supervision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2021), 2613-2622.

[10] Y. Lu, Y. Chen, D. Zhao, B. Liu, Z. Lai, and J. Chen, “CNN-G: Convolutional Neural Network Combined With Graph for Image Seg-mentation With Theoretical Analysis,” IEEE Transactions on Cognitive And Developmental Systems 13, no. 3 (2020): 631-644, https://doi.org/10.1109/tcds.2020.2998497.

[11] Y. Wang, H. Wang, Y. Shen, et al., “Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2022),4248-4257.

[12] X. Li, L. Yu, H. Chen, C. W. Fu, and P. A. Heng, “Semi-Supervised Skin Lesion Segmentation Via Transformation Consistent Self-Ensembling Model,” arXiv preprint arXiv:1808. 03887 (2018).

[13] X. Li, L. Yu, H. Chen, C. W. Fu, L. Xing, and P. A. Heng, “Transformation-Consistent Self-Ensembling Model for Semisupervised Medical Image Segmentation,” IEEE transactions on neural networks and learning systems 32, no. 2 (2020): 523-534, https://doi.org/10.1109/tnnls.2020.2995319.

[14] X. Luo, J. Chen, T. Song, and G. Wang, “Semi-Supervised Medical Image Segmentation Through Dual-Task Consistency,” in Proceedings of the AAAI Conference on Artificial Intelligence 35, (2021), 8801-8809.

[15] B. Huang, A. Nguyen, S. Wang, et al., “Simultaneous Depth Esti-mation and Surgical Tool Segmentation in Laparoscopic Images,” IEEE Transactions on Medical Robotics and Bionics 4, no. 2 (2022): 335-338, https://doi.org/10.1109/tmrb.2022.3170215.

[16] L. Yang, L. Qi, L. Feng, W. Zhang, and Y. Shi, “Revisiting Weak-To-Strong Consistency in Semi-Supervised Semantic Segmentation,” in Proceedings of the Conference on Computer Vision and Pattern Recogni-tion, (2023), 7236-7246.

[17] H. Lee, “Engineering In Vitro Models: Bioprinting of Organoids With Artificial Intelligence,” Cyborg Bionic Systems 4 (2023): 0018, https://doi.org/10.34133/cbsystems.0018.

[18] H. Wang, T. Fu, Y. Du, et al., “Scientific Discovery in the Age of Artificial Intelligence,” Nature 620, no. 7972 (2023): 47-60, https://doi.org/10.1038/s41586-023-06221-2.

[19] X. Li, Z. Zhou, K. Wang, et al. “An Improved Image Super-Resolution Algorithm for Percutaneous Endoscopic Lumbar Dis-cectomy,” , in International Conference on Cognitive Systems and Signal Processing (Springer, 2023), 149-160.

[20] Z. Chen, Q. Liang, Z. Wei, et al., “An Overview of In Vitro Biological Neural Networks for Robot Intelligence,” Cyborg Bionic Systems 4 (2023): 0001, https://doi.org/10.34133/cbsystems.0001.

[21] G. Zhang, J. Su, F. Du, X. Zhang, Y. Li, and R. Song, “Composite Continuum Robots: Accurate Modeling and Model Reduction,” Inter-national Journal of Mechanical Sciences 276 (2024): 109342, https://doi.org/10.1016/j.ijmecsci.2024.109342.

[22] F. Du, G. Zhang, Y. Xu, Y. Lei, R. Song, and Y. Li, “Continuum Robots: Developing Dexterity Evaluation Algorithms Using Efficient Inverse Kinematics,” Measurement 216 (2023): 112925, https://doi.org/10.1016/j.measurement.2023.112925.

[23] T. Brown, B. Mann, N. Ryder, et al., “Language Models Are Few-Shot Learners,” Advances in Neural Information Processing Systems 33 (2020): 1877-1901.

[24] H. Touvron, T. Lavril, G. Izacard, et al., “Llama: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302. 13971 (2023).

[25] A. Kirillov, E. Mintun, N. Ravi, et al. “Segment Anything ” , in Proceedings of the IEEE International Conference on Computer Vision, (2023), 4015-4026.

[26] Y. Zhang, Y. Cheng, and Y. Qi, “SemiSAM: Exploring SAM for Enhancing Semi-Supervised Medical Image Segmentation With Extremely Limited Annotations,” arXiv preprint arXiv:2312. 06316 (2023).

[27] X. Lu, L. Jiao, L. Li, et al., “Weak-To-Strong Consistency Learning for Semisupervised Image Segmentation,” IEEE Transactions on Geo-science and Remote Sensing 61 (2023): 1-15, https://doi.org/10.1109/tgrs.2023.3272552.

[28] S. Yun, D. Han, S. J. Oh, S. Chun,J. Choe, and Y. Yoo, “Cutmix: Regularization Strategy to Train Strong Classifiers With Localizable Features,” in Proceedings of the IEEE International Conference on Computer Vision, (2019), 6023-6032.

[29] L. C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-Decoder With Atrous Separable Convolution for Semantic Image Segmentation,” in Proceedings of the European Conference on Computer Vision (ECCV, 2018), 801-818.

[30] S. Gul, M. S. Khan, A. Bibi, A. Khandakar, M. A. Ayari, and M. E. Chowdhury, “Deep Learning Techniques for Liver and Liver Tumor Segmentation: A Review,” Computers in Biology and Medicine 147 (2022): 105620, https://doi.org/10.1016/j.compbiomed.2022.105620.

[31] K. He, X. Zhang,S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778.

[32] K. K. Brar, B. Goyal, A. Dogra, et al., “Image Segmentation Review: Theoretical Background and Recent Advances,” Information Fusion 114 (2024): 102608, https://doi.org/10.1016/j.inffus.2024.102608.

[33] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing 13, no. 4 (2004): 600-612.

[34] R. Zhang, P. Isola, A. A. Efros,E. Shechtman, and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), 586-595.

[35] R. Jiao, Y. Zhang, L. Ding, et al., “Learning With Limited Annota-tions: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation,” Computers in Biology and Medicine 169 (2023): 107840, https://doi.org/10.1016/j.compbiomed.2023.107840.

[36] R. Jiao, Y. Zhang, L. Ding, et al., “Learning With Limited Annota-tions: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation,” Computers in Biology and Medicine 169 (2024): 107840, https://doi.org/10.1016/j.compbiomed.2023.107840.

[37] Y. Liu, T. Zhen, Y. Fu, Y. Wang, A. Han, et al., “AI-Powered Seg-mentation of Invasive Carcinoma Regions in Breast Cancer Immuno-histochemical Whole-Slide Images,” Cancers 16, no. 1 (2023): 167, https://doi.org/10.3390/cancers16010167.

[38] M. Allan, A. Shvets, T. Kurmann, et al., “2017 Robotic Instrument Segmentation Challenge,” arXiv preprint arXiv:1902. 06426 (2019).

[39] M. Allan, S. Kondo, S. Bodenstedt, et al., “2018 Robotic Scene Segmentation Challenge,” arXiv preprint arXiv:2001. 11190 (2020).

Funding

National Key R and D Program of China(Grant 2023YFB4706300)