The importance of machine learning in autonomous actions for surgical decision making

Abstract

Surgery faces a paradigm shift since it has developed rapidly in recent decades, becoming a high-tech discipline. Increasingly powerful technological developments such as modern operating rooms, featuring digital and interconnected equipment and novel imaging as well as robotic procedures, provide several data sources resulting in a huge potential to improve patient therapy and surgical outcome by means of Surgical Data Science. The emerging field of Surgical Data Science aims to improve the quality of surgery through acquisition, organization, analysis, and modeling of data, in particular using machine learning (ML). An integral part of surgical data science is to analyze the available data along the surgical treatment path and provide a context-aware autonomous action by means of ML methods. Autonomous actions related to surgical decision-making include preoperative decision support, intraoperative assistance functions, as well as robot-assisted actions. The goal is to democratize surgical skills and enhance the collaboration between surgeons and cyber-physical systems by quantifying surgical experience and making it accessible to machines, thereby improving patient therapy and outcome. The article introduces basic ML concepts as enablers for autonomous actions in surgery, highlighting examples for such actions along the surgical treatment path.

Keywords

Surgical data science / robot-assisted surgery / artificial intelligence in surgery / cognitive surgical robotics / computer-aided surgery

References

[1] Nepogodiev D,Biccard B.Global burden of postoperative death.The Lancet2019;393:401

[2] Weiser TG,Thompson KD.An estimation of the global volume of surgery: a modelling strategy based on available data.Lancet2008;372:139-44

[3] Birkmeyer JD,Finlayson EV.Hospital volume and surgical mortality in the United States.N Engl J Med2002;346:1128-37

[4] Tomašev N,Rae JW.A clinically applicable approach to continuous prediction of future acute kidney injury.Nature2019;572:116-9 PMCID:PMC6722431

[5] Muti HS,Keller G.Development and validation of deep learning classifiers to detect Epstein-Barr virus and microsatellite instability status in gastric cancer: a retrospective multicentre cohort study.Lancet Digital Health2021;3:e654-64 PMCID:PMC8460994

[6] Esteva A,Novoa RA.Dermatologist-level classification of skin cancer with deep neural networks.Nature2017;542:115-8 PMCID:PMC8382232

[7] Maier-Hein L,Sarikaya D.Surgical data science - from concepts toward clinical translation.Med Image Anal2022;76:102306

[8] Maier-Hein L,Speidel S.Surgical data science for next-generation interventions.Nat Biomed Eng2017;1:691-6

[9] Russell S. Artificial intelligence, global edition. Available from: https://elibrary.pearson.de/book/99.150005/9781292401171 [Last accessed on 14 Apr 2022]

[10] Kaelbling LP,Moore AW.Reinforcement learning: a survey.Journal of Artificial Intelligence Research1996;4:237

[11] Schmidhuber J.Deep learning in neural networks: an overview.Neural Netw2015;61:85-117

[12] Lecun Y,Denker JS.Backpropagation applied to handwritten zip code recognition.Neural Computation1989;1:541-51

[13] Waibel A,Hinton G,Lang K.Phoneme recognition using time-delay neural networks.IEEE Trans Neural Netw1989;37:328-39

[14] Ross T,Vemuri A.Exploiting the potential of unlabeled endoscopic video data with self-supervised learning.Int J Comput Assist Radiol Surg2018;13:925-33

[15] Zhai X,Kolesnikov A. S4L: self-supervised semi-supervised learning. Available from: https://arxiv.org/abs/1905.03670 [Last accessed on 14 Apr 2022]

[16] Wang Y,Kwok JT.Generalizing from a few examples: a survey on few-shot learning.ACM Comput Surv2021;53:1-34

[17] Shrestha A.Review of deep learning algorithms and architectures.IEEE Access2019;7:53040-65

[18] Kendall A. What uncertainties do we need in bayesian deep learning for computer vision? Available from: https://papers.nips.cc/paper/2017/hash/2650d6089a6d640c5e85b2b88265dc2b-Abstract.html [Last accessed on 14 Apr 2022]

[19] Hüllermeier E.Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.Mach Learn2021;110:457-506

[20] Adler TJ,Vemuri A.Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks.Int J Comput Assist Radiol Surg2019;14:997-1007

[21] Ardizzone L,Rother C. Analyzing inverse problems with invertible neural networks. Available from: https://arxiv.org/abs/1808.04730 [Last accessed on 14 Apr 2022]

[22] Barredo Arrieta A,Del Ser J.Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI.Information Fusion2020;58:82-115

[23] Hashimoto DA,Rus D.Artificial intelligence in surgery: promises and perils.Ann Surg2018;268:70-6 PMCID:PMC5995666

[24] Mckinney SM,Godbole V.International evaluation of an AI system for breast cancer screening.Nature2020;577:89-94

[25] Kudo SE,Villard B.Artificial intelligence system to determine risk of T1 colorectal cancer metastasis to lymph node.Gastroenterology2021;160:1075-1084.e2

[26] Mai RY,Bai T.Artificial neural network model for preoperative prediction of severe liver failure after hemihepatectomy in patients with hepatocellular carcinoma.Surgery2020;168:643-52

[27] Que SJ,Qing-Zhong .Application of preoperative artificial neural network based on blood biomarkers and clinicopathological parameters for predicting long-term survival of patients with gastric cancer.World J Gastroenterol2019;25:6451-64 PMCID:PMC6881508

[28] Rice TW,Ishwaran H.Worldwide Esophageal Cancer Collaboration InvestigatorsPrecision surgical therapy for adenocarcinoma of the esophagus and esophagogastric junction.J Thorac Oncol2019;14:2164-75 PMCID:PMC6876319

[29] Zou FW,Liu CY,Hu CH.Concordance study between IBM watson for oncology and real clinical practice for cervical cancer patients in China: a retrospective analysis.Front Genet2020;11:200 PMCID:PMC7105853

[30] Powles J.Google deepmind and healthcare in an age of algorithms.Health Technol (Berl)2017;7:351-67 PMCID:PMC5741783

[31] Murdoch TB.The inevitable application of big data to health care.JAMA2013;309:1351-2

[32] Healey T,Healey J,Saleh KJ.Improving operating room efficiency, part 1: general managerial and preoperative strategies.JBJS Rev2015;3:e3

[33] Stahl JE,Daily B.Reorganizing patient care and workflow in the operating room: a cost-effectiveness study.Surgery2006;139:717-28

[34] Marjamaa RA,Hirvensalo EJ.What is the best workflow for an operating room?.Health Care Manag Sci2009;12:142-6

[35] Bercker S,Hokema F.[Effects of overlapping induction on the utilization of complex operating structures: estimation using the practical application of a simulation model].Anaesthesist2013;62:440-6

[36] Souders CP,Wood LN.Reducing operating room turnover time for robotic surgery using a motor racing pit stop model.World J Surg2017;41:1943-9 PMCID:PMC6010351

[37] Tanzi L,Vezzetti E.Intraoperative surgery room management: A deep learning perspective.Int J Med Robot2020;16:1-12

[38] Berthet-Rayne P,King H. Hubot: A three state Human-Robot collaborative framework for bimanual surgical tasks based on learned models. Available from: https://ieeexplore.ieee.org/document/7487198 [Last accessed on 14 Apr 2022]

[39] Zhao B,Urman RD.A machine learning approach to predicting case duration for robot-assisted surgery.J Med Syst2019;43:32

[40] Wagner M,Kenngott HG.A learning robot for cognitive camera control in minimally invasive surgery.Surg Endosc2021;35:5365-74 PMCID:PMC8346448

[41] Katić D,Görtler J.Context-aware Augmented Reality in laparoscopic surgery.Comput Med Imaging Graph2013;37:174-82

[42] Kennedy-Metz LR,Torralba A.Computer vision in the operating room: opportunities and caveats.IEEE Trans Med Robot Bionics2021;3:2-10 PMCID:PMC7908934

[43] Garrow CR,Li L.Machine learning for surgical phase recognition: a systematic review.Ann Surg2021;273:684-93

[44] Hashimoto DA,Volkov M,Meireles OR.Artificial intelligence for intraoperative video analysis: machine learning’s role in surgical education.Journal of the American College of Surgeons2017;225:4

[45] Twinanda AP,Mutter D,de Mathelin M.EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos.IEEE Trans Med Imaging2017;36:86-97

[46] Wagner M,Kisilenko A. Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark. Available from: https://arxiv.org/ftp/arxiv/papers/2109/2109.14956.pdf

[47] Moonesinghe SR,Das P,Grocott MP.Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: qualitative systematic review.Anesthesiology2013;119:959-81

[48] Suliburk JW,Pirko CJ.Analysis of human performance deficiencies associated with surgical adverse events.JAMA Netw Open2019;2:e198067 PMCID:PMC6669897

[49] Flin R,Yule S.How do surgeons make intraoperative decisions?.Qual Saf Health Care2007;16:235-9 PMCID:PMC2464983

[50] Glance LG,Neuman MD.Redesigning surgical decision making for high-risk patients.N Engl J Med2014;370:1379-81 PMCID:PMC4344120

[51] Garg AX,McDonald H.Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review.JAMA2005;293:1223-38

[52] Harangi B,Lampé R. Recognizing ureter and uterine artery in endoscopic images using a convolutional neural network. Available from: https://www.semanticscholar.org/paper/Recognizing-Ureter-and-Uterine-Artery-in-Endoscopic-Harangi-Hajdu/b9956febca47a364eb23b6cc65bccfec06206509 [Last accessed on Apr 2022]

[53] Mascagni P,Alapatt D.Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning.Ann Surg2020;

[54] Strasberg SM.Rationale and use of the critical view of safety in laparoscopic cholecystectomy.J Am Coll Surg2010;211:132-8

[55] Quellec G,Cazuguel G. Real-time retrieval of similar videos with application to computer-aided retinal surgery. Available from: https://ieeexplore.ieee.org/document/6091107 [Last accessed on 14 Apr 2022]

[56] Li Y,Liu Y,Giannarou S.Context aware decision support in neurosurgical oncology based on an efficient classification of endomicroscopic data.Int J Comput Assist Radiol Surg2018;13:1187-99 PMCID:PMC6096753

[57] Halicek M,Wang X.Optical biopsy of head and neck cancer using hyperspectral imaging and convolutional neural networks.Proc SPIE Int Soc Opt Eng2018;10469:104690X PMCID:PMC6123819

[58] Hou F,Yang Z. Automatic identification of metastatic lymph nodes in OCT images. Available from: https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10867/108673G/Automatic-identification-of-metastatic-lymph-nodes-in-OCT-images/10.1117/12.2511588.short?SSO=1 [Last accessed on 14 Apr 2022]

[59] Ritschel K,Winter S.Brain tumor classification on intraoperative contrast-enhanced ultrasound.Int J Comput Assist Radiol Surg2015;10:531-40

[60] Maier-Hein L,Sarikaya D. Surgical data science - from concepts toward clinical translation. Available from: https://arxiv.org/abs/2011.02284 [Last accessed on 14 Apr 2022]

[61] Cabitza F,Gensini GF.Unintended consequences of machine learning in medicine.JAMA2017;318:517-8

[62] Safdar NM,Meltzer CC.Ethical considerations in artificial intelligence.Eur J Radiol2020;122:108768

[63] Lundberg SM,Vavilala MS.Explainable machine-learning predictions for the prevention of hypoxaemia during surgery.Nat Biomed Eng2018;2:749-60 PMCID:PMC6467492

[64] Navarrete-Welton AJ.Current applications of artificial intelligence for intraoperative decision support in surgery.Front Med2020;14:369-81

[65] Moustris GP,Deliparaschos KM.Evolution of autonomous and semi-autonomous robotic surgical systems: a review of the literature.Int J Med Robot2011;7:375-92

[66] Harris SJ,Mei Q.The Probot--an active robot for prostate resection.Proc Inst Mech Eng H1997;211:317-25

[67] Hutchinson K,Bhatia H. A reactive autonomous camera system for the RAVEN II surgical robot. Available from: https://arxiv.org/abs/2010.04785 [Last accessed on 14 Apr 2022]

[68] Rivas-blanco I,Perez-del-pulgar CJ,Fraile JC.Smart cable-driven camera robotic assistant.IEEE Trans Human-Mach Syst2018;48:183-96

[69] Mayer H,Wierstra D. A System for robotic heart surgery that learns to tie knots using recurrent neural networks. Available from: https://ieeexplore.ieee.org/document/4059310 [Last accessed on 14 Apr 2022]

[70] Padoy N. Human-machine collaborative surgery using learned models. Available from: https://ieeexplore.ieee.org/document/5980250 [Last accessed on 14 Apr 2022]

[71] Knoll A,Staub C.Selective automation and skill transfer in medical robotics: a demonstration on surgical knot-tying.Int J Med Robot2012;8:384-97

[72] van den Berg J,Duckworth D. Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations. Available from: https://ieeexplore.ieee.org/document/5509621 [Last accessed on 14 Apr 2022]

[73] Mikada T,Kawase T,Kawashima K.Suturing support by human cooperative robot control using deep learning.IEEE Access2020;8:167739-46

[74] Lu B,Jin YM. A learning-driven framework with spatial optimization for surgical suture thread reconstruction and autonomous grasping under multiple topologies and environmental noises. Available from: https://arxiv.org/abs/2007.00920 [Last accessed on 14 Apr 2022]

[75] Schwaner KL,Jensen PT. Autonomous needle manipulation for robotic surgical suturing based on skills learned from demonstration. Available from: https://ieeexplore.ieee.org/document/9551569 [Last accessed on 14 Apr 2022]

[76] Chiu ZY,Funk EK. Bimanual regrasping for suture needles using reinforcement learning for rapid motion planning. Available from: https://arxiv.org/abs/2011.04813 [Last accessed on 14 Apr 2022]

[77] Peters JH,Swanstrom LL.Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery.Surgery2004;135:21-7

[78] Thananjeyan B,Krishnan S. Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning. Available from: https://ieeexplore.ieee.org/document/7989275 [Last accessed on 14 Apr 2022]

[79] Nguyen ND,Nahavandi S. Manipulating soft tissues by deep reinforcement learning for autonomous robotic surgery. Available from: https://ieeexplore.ieee.org/document/8836924 [Last accessed on 14 Apr 2022]

[80] Pedram SA,Shin C. Toward synergic learning for autonomous manipulation of deformable tissues via surgical robots: an approximate q-learning approach. Available from: https://arxiv.org/abs/1910.03398 [Last accessed on 14 Apr 2022]

[81] Shin C,Pedram SA. Autonomous tissue manipulation via surgical robot using learning based model predictive control. Available from: https://arxiv.org/abs/1902.01459 [Last accessed on 14 Apr 2022]

[82] Xu W,Lau HYK. Automate surgical tasks for a flexible serpentine manipulator via learning actuation space trajectory from demonstration. Available from: https://ieeexplore.ieee.org/document/7487640 [Last accessed on 14 Apr 2022]

[83] Barragan JA,Yu D. SACHETS: semi-autonomous cognitive hybrid emergency teleoperated suction. Available from: https://ieeexplore.ieee.org/document/9515517 [Last accessed on 14 Apr 2022]

[84] Nichols KA.A framework for multilateral manipulation in surgical tasks.IEEE Trans Automat Sci Eng2016;13:68-77

[85] Kehoe B,Mahler J. Autonomous multilateral debridement with the Raven surgical robot. Available from: http://people.eecs.berkeley.edu/~pabbeel/papers/2014-ICRA-multilateral-ravens.pdf [Last accessed on 14 Apr 2022]

[86] Kim JW,Gehlbach P.Towards autonomous eye surgery by combining deep imitation learning with optimal control.Proc Mach Learn Res2021;155:2347-58 PMCID:PMC8549631

[87] Keller B,Zhou K.Optical coherence tomography-guided robotic ophthalmic microsurgery via reinforcement learning from demonstration.IEEE Trans Robot2020;36:1207-18

[88] Baek D,Kim H. Path planning for automation of surgery robot based on probabilistic roadmap and reinforcement learning. Available from: https://ieeexplore.ieee.org/document/8441801 [Last accessed on 14 Apr 2022]

[89] Shademan A,Opfermann JD,Krieger A.Supervised autonomous robotic soft tissue surgery.Sci Transl Med2016;8:337ra64

[90] Saeidi H,Kam M.Autonomous robotic laparoscopic surgery for intestinal anastomosis.Sci Robot2022;7:eabj2908 PMCID:PMC8992572

[91] Ning G,Zhang X.Force-guided autonomous robotic ultrasound scanning control method for soft uncertain environment.Int J Comput Assist Radiol Surg2021;16:2189-99

[92] Butler D.Translational research: crossing the valley of death.Nature2008;453:840-2