LungSurg: A Generative AI System for Segmentation and Phase Classification in Thoracoscopic Lobectomy

Abstract

The integration of artificial intelligence (AI) into surgical practices is advancing towards greater intelligence and precision. This study assesses the potential of AI in video-assisted thoracoscopic surgery (VATS) lobectomy for lung cancer by developing an AI system named LungSurg. LungSurg comprises two interconnected networks: a segmentation network for identifying intrathoracic anatomy and surgical instruments, and a classification network for recognizing surgical phases. We prospectively collected 222 VATS lobectomy videos from eight centers, generating over 32,000 annotations and more than one million frames with phase information. In external validation, the segmentation network achieved mean Average precision scores of 0.745 for the left lung and 0.726 for the right lung across various instruments and anatomical structures. The classification network demonstrated Top-1 and Top-3 accuracies of 71.5% and 88.0%, respectively, in identifying 14 surgical phases. Comparative experiments revealed that LungSurg performed comparably to senior surgeons in anatomical identification and surpassed them in sensitivity. In addition, an educational study showed that surgical residents trained with LungSurg significantly improved their anatomical identification and phase classification skills compared to those using conventional methods. These results indicate that LungSurg accurately analyzes VATS lobectomy procedures, highlighting the feasibility and potential of AI-driven tools in enhancing thoracic surgical practices.

Keywords

deep learning / generative artificial intelligence / lobectomy / surgical education / thoracic surgery

References

[1] H. Sung, J. Ferlay, R. L. Siegel, et al., “Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries,” CA: A Cancer Journal for Clinicians 71, no. 3 (2021): 209–249.

[2] T. A. D'Amico, “The Video-Assisted Thoracoscopic or Open Lobectomy (VIOLET) Trial: The Final Chapter to This Epic,” Journal of Thoracic and Cardiovascular Surgery 166, no. 1 (2023): 265–267.

[3] T. A. Haidari, L. J. Nayahangan, F. Bjerrum, et al., “Consensus on Technical Procedures for Simulation-Based Training in Thoracic Surgery: An International Needs Assessment,” European Journal of Cardio-Thoracic Surgery 63, no. 4 (2023): ezad058.

[4] A. Vieira, E. Bourdages-Pageau, K. Kennedy, and P. A. Ugalde, “The Learning Curve on Uniportal Video-Assisted Thoracic Surgery: An Analysis of Proficiency,” Journal of Thoracic and Cardiovascular Surgery 159, no. 6 (2020): 2487–2495.e2.

[5] S. Grossi, M. Cattoni, N. Rotolo, and A. Imperatori, “Video-Assisted Thoracoscopic Surgery Simulation and Training: A Comprehensive Literature Review,” BMC Medical Education 23, no. 1 (2023): 535.

[6] Y. Li, N. Raison, S. Ourselin, T. Mahmoodi, P. Dasgupta, and A. Granados, “AI Solutions for Overcoming Delays in Telesurgery and Telementoring to Enhance Surgical Practice and Education,” Journal of Robotic Surgery 18, no. 1 (2024): 403.

[7] D. A. Hashimoto, G. Rosman, D. Rus, and O. R. Meireles, “Artificial Intelligence in Surgery: Promises and Perils,” Annals of Surgery 268, no. 1 (2018): 70–76.

[8] T. M. Ward, D. A. Hashimoto, Y. Ban, et al., “Automated Operative Phase Identification in Peroral Endoscopic Myotomy,” Surgical Endoscopy 35, no. 7 (2021): 4008–4015.

[9] R. Garcia Nespolo, D. Yi, E. Cole, N. Valikodath, C. Luciano, and Y. I. Leiderman, “Evaluation of Artificial Intelligence-Based Intraoperative Guidance Tools for Phacoemulsification Cataract Surgery,” JAMA Ophthalmology 140, no. 2 (2022): 170–177.

[10] P. Mascagni, A. Vardazaryan, D. Alapatt, et al., “Artificial Intelligence for Surgical Safety: Automatic Assessment of the Critical View of Safety in Laparoscopic Cholecystectomy Using Deep Learning,” Annals of Surgery 275, no. 5 (2022): 955–961.

[11] S. Wu, Z. Chen, R. Liu, et al., “SurgSmart: An Artificial Intelligent System for Quality Control in Laparoscopic Cholecystectomy: An Observational Study,” International Journal of Surgery 109, no. 5 (2023): 1105–1114.

[12] Z. Chen, J. An, S. Wu, et al., “Surgesture: A Novel Instrument Based on Surgical Actions for Objective Skill Assessment,” Surgical Endoscopy 36, no. 8 (2022): 6113–6121.

[13] Z. Chen, D. Yang, A. Li, et al., “Decoding Surgical Skill: An Objective and Efficient Algorithm for Surgical Skill Classification Based on Surgical Gesture Features-Experimental Studies,” International Journal of Surgery 110, no. 3 (2024): 1441–1449.

[14] Z. Chen, Y. Zhang, Z. Yan, et al., “Artificial Intelligence Assisted Display in Thoracic Surgery: Development and Possibilities,” Journal of Thoracic Disease 13, no. 12 (2021): 6994–7005.

[15] O. Elharrouss, N. Almaadeed, S. Al-Maadeed, and Y. Akbari, “Image Inpainting: A Review,” Neural Processing Letters 51 (2020): 2007–2028.

[16] S. Wu, M. Tang, J. Liu, et al., “Impact of an AI-Based Laparoscopic Cholecystectomy Coaching Program on the Surgical Performance: A Randomized Controlled Trial,” International Journal of Surgery 110, no. 12 (2024): 7816–7823.

[17] P. Mascagni, D. Alapatt, A. Lapergola, et al., “Early-Stage Clinical Evaluation of Real-Time Artificial Intelligence Assistance for Laparoscopic Cholecystectomy,” British Journal of Surgery 111, no. 1 (2024): znad353.

[18] M. Kawamura, Y. Endo, A. Fujinaga, et al., “Development of an Artificial Intelligence System for Real-Time Intraoperative Assessment of the Critical View of Safety in Laparoscopic Cholecystectomy,” Surgical Endoscopy 37, no. 11 (2023): 8755–8763.

[19] S. Sai, A. Gaur, R. Sai, V. Chamola, M. Guizani, and J. J. Rodrigues, “Generative AI for Transformative Healthcare: A Comprehensive Study of Emerging Models, Applications, Case Studies and Limitations,” IEEE Access 2 (2024): 31078–31106.

[20] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked Autoencoders Are Scalable Vision Learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR, 2022), 16000–16009.

[21] Y. Chen, X.-H. Yang, Z. Wei, et al., “Generative Adversarial Networks in Medical Image Augmentation: A Review,” Computers in Biology and Medicine 144 (2022): 105382.

[22] M. S. Hameed, S. Laplante, C. Masino, et al., “What Is the Educational Value and Clinical Utility of Artificial Intelligence for Intraoperative and Postoperative Video Analysis? A Survey of Surgeons and Trainees,” Surgical Endoscopy 37, no. 12 (2023): 9453–9460.

[23] M. U. Khalid, S. Laplante, C. Masino, et al., “Use of Artificial Intelligence for Decision-Support to Avoid High-Risk Behaviors During Laparoscopic Cholecystectomy,” Surgical Endoscopy 37, no. 12 (2023): 9467–9475.

[24] J. Jam, C. Kendrick, K. Walker, V. Drouard, J. G. Hsu, and M. H. Yap, “A Comprehensive Review of Past and Present Image Inpainting Methods,” Computer Vision and Image Understanding 203 (2021): 103147.

[25] X. Zhang, D. Zhai, T. Li, Y. Zhou, and Y. Lin, “Image Inpainting Based on Deep Learning: A Review,” Information Fusion 90 (2023): 74–94.

[26] J. Jiao, C. Yin, and F. Teng, “W-Net: Deep Convolutional Network With Gray-Level Co-Occurrence Matrix and Hybrid Loss Function for Hyperspectral Image Classification,” in Advanced Intelligent Computing Technology and Applications, (Springer, 2023), 112–124.

[27] B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman, “LabelMe: A Database and Web-Based Tool for Image Annotation,” International Journal of Computer Vision 77 (2008): 157–173.

[28] M. K. Ferguson and C. Bennett, “Identification of Essential Components of Thoracoscopic Lobectomy and Targets for Simulation,” Annals of Thoracic Surgery 103, no. 4 (2017): 1322–1329.

[29] D. S. Bryan, M. K. Ferguson, M. B. Antonoff, et al., “Consensus for Thoracoscopic Left Upper Lobectomy-Essential Components and Targets for Simulation,” Annals of Thoracic Surgery 112, no. 2 (2021): 436–442.

[30] P. A. Erwin, A. C. Lee, U. Ahmad, et al., “Consensus for Thoracoscopic Lower Lobectomy: Essential Components and Targets for Simulation,” Annals of Thoracic Surgery 114, no. 5 (2022): 1895–1901.

[31] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, (CVPR, 2017), 2961–2969.

[32] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-Local Neural Networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, (CVPR, 2018), 7794–7803.

[33] A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani, “Bottleneck Transformers for Visual Recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, (CVPR, 2021), 16519–16529.

[34] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection With Region Proposal Networks,” preprint, arXiv, June 4, 2015.

[35] A. Howard, M. Sandler, G. Chu, et al., “Searching for Mobilenetv3,” in Proceedings of the IEEE International Conference on Computer Vision, (ICCV, 2019), 1314–1324.

[36] P. Zhou, W. Shi, J. Tian, et al., “Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ed. K. Erk and N. A. Smith, (Association for Computational Linguistics, 2016), 207–212.

RIGHTS & PERMISSIONS

2026 The Author(s). MedComm published by Sichuan International Medical Exchange & Promotion Association (SCIMEA) and John Wiley & Sons Australia, Ltd.