Real-time robust liver and gallbladder segmentation during laparoscopic cholecystectomy using convolutional neural networks: an analysis

Abstract

Aim: Images in different laparoscopic cholecystectomy datasets are acquired using various camera models, parameters, and settings, with the annotation methods varying by institution. These factors result in inconsistent inference performance of the network model. This study aims to identify the optimal network model architecture for liver and gallbladder segmentation from several options. Then, the performance and robustness of the optimal network model are evaluated using an independent dataset that is not included in the training.

Methods: The public dataset, CholecSeg8k, was utilized as the input for the network model training, validation, and testing. A local private dataset from KPJ Damansara Hospital, Selangor, Malaysia, was used for testing purposes only. For the implementation of liver and gallbladder segmentation, segmentation models, a public Python library was employed.

Results: Among the experiments, highly accurate liver and gallbladder segmentation results were achieved using the feature pyramid network (FPN) architecture as the network model, with the Inception-ResNet-v2 architecture as the network backbone. The best-trained network model resulted in a loss of 0.070955, a mean intersection over union (IoU) score of 0.95896, and a mean F1-score of 0.9773 on the test set. However, visualized results for the private dataset contained considerable false-negative areas.

Conclusion: The proposed automated technique has the potential to serve as an alternative to the conventional indocyanine green injection along with near-infrared fluorescence imaging (ICG-NIRF)-based method for liver and gallbladder segmentation during laparoscopic cholecystectomy. Future work will focus on enhancing the results of the private dataset. Additionally, a surgeon-assistant robotic arm that will use the liver and gallbladder segmentation results for camera steering will be analyzed.

Keywords

Artificial intelligence / convolutional neural network / deep learning / computer vision / liver and gallbladder segmentation / laparoscopic cholecystectomy

References

[1] Fu Z,Zhang C.The future of endoscopic navigation: a review of advanced endoscopic vision technology.IEEE Access2021;9:41144-67

[2] Wang X,Ishizawa T.Consensus guidelines for the use of fluorescence imaging in hepatobiliary surgery.Ann Surg2021;274:97-106

[3] Wendler T,Navab N.How molecular imaging will enable robotic precision surgery: the role of artificial intelligence, augmented reality, and navigation.Eur J Nucl Med Mol Imaging2021;48:4201-24 PMCID:PMC8566413

[4] Zaffino P,De Momi E.A review on advances in intra-operative imaging for surgery and therapy: imagining the operating room of the future.Ann Biomed Eng2020;48:2171-91

[5] Madani A,Altieri MS.Artificial intelligence for intraoperative guidance: using semantic segmentation to identify surgical anatomy during laparoscopic cholecystectomy.Ann Surg2022;276:363-9 PMCID:PMC8186165

[6] Scheikl PM,Kisilenko A.Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery.Curr Dir Biomed Eng2020;6:20200016

[7] Mascagni P,Alapatt D.Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning.Ann Surg2022;275:955-61

[8] Mascagni P,Lapergola A.Early-stage clinical evaluation of real-time artificial intelligence assistance for laparoscopic cholecystectomy.Br J Surg2024;111:znad353

[9] Mascagni P,Garcia A. Surgical data science for safe cholecystectomy: a protocol for segmentation of hepatocystic anatomy and assessment of the critical view of safety. arXiv. [Preprint.] Sep 20, 2021. [accessed on 2024 Sep 21]. Available from: https://doi.org/10.48550/arXiv.2106.10916.

[10] Alt B,Katic D.LapSeg3D: weakly supervised semantic segmentation of point clouds representing laparoscopic scenes. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2022 Oct 23-27; Kyoto, Japan. IEEE; 2022. pp. 5265-70.

[11] Mascagni P,Sestini L.Computer vision in surgery: from potential to clinical value.NPJ Digit Med2022;5:163 PMCID:PMC9616906

[12] Maier-Hein L,Sarikaya D.Surgical data science - from concepts toward clinical translation.Med Image Anal2022;76:102306 PMCID:PMC9135051

[13] Hong WY,Kuo YH,Chang WL. CholecSeg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on Cholec80. arXiv. [Preprint.] Dec 23, 2020. [accessed on 2024 Sep 21]. Available from: https://doi.org/10.48550/arXiv.2012.12453.

[14] Iakubovskii P. Segmentation_models. 2019. Available from: https://github.com/qubvel/segmentation_models. [Last accessed on 21 Sep 2024]

[15] He K,Dollár P.Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV); 2017 Oct 22-29; Venice, Italy. IEEE; 2017. pp. 2980-8.

[16] Lin TY,Girshick R,Hariharan B. Feature pyramid networks for object detection. arXiv. [Preprint.] Apr 19, 2017. [accessed on 2024 Sep 21]. Available from: https://doi.org/10.48550/arXiv.1612.03144.

[17] Szegedy C,Ioffe S,Wojna Z.Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 Jun 27-30; Las Vegas, USA. IEEE; 2016. pp. 2818-26.

[18] He K,Ren S.Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 Jun 27-30; Las Vegas, USA. IEEE; 2016. pp. 770-8.

[19] Szegedy C,Vanhoucke V.Inception-v4, inception-ResNet and the impact of residual connections on learning. In: 31st AAAI Conference on Artificial Intelligence. 2017. pp. 4278-84.

[20] Zhao H,Qi X,Jia J.Pyramid scene parsing network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21-26; Honolulu, USA. IEEE; 2017. pp. 6230-9.