Applying artificial intelligence to big data in hepatopancreatic and biliary surgery: a scoping review

Abstract

Aim: Artificial Intelligence (AI) and its applications in healthcare are rapidly developing. The healthcare industry generates ever-increasing volumes of data that should be used to improve patient care. This review aims to examine the use of AI and its applications in hepatopancreatic and biliary (HPB) surgery, highlighting studies leveraging large datasets.

Methods: A PRISMA-ScR compliant scoping review using Medline and Google Scholar databases was performed (5^th August 2022). Studies focusing on the development and application of AI to HPB surgery were eligible for inclusion. We undertook a conceptual mapping exercise to identify key areas where AI is under active development for use in HPB surgery. We considered studies and concepts in the context of patient pathways - before surgery (including diagnostics), around the time of surgery (supporting interventions) and after surgery (including prognostication).

Results: 98 studies were included. Most studies were performed in China or the USA (n = 45). Liver surgery was the most common area studied (n = 51). Research into AI in HPB surgery has increased rapidly in recent years, with almost two-thirds published since 2019 (61/98). Of these studies, 11 have focused on using “big data” to develop and apply AI models. Nine of these studies came from the USA and nearly all focused on the application of Natural Language Processing. We identified several critical conceptual areas where AI is under active development, including improving preoperative optimization, image guidance and sensor fusion-assisted surgery, surgical planning and simulation, natural language processing of clinical reports for deep phenotyping and prediction, and image-based machine learning.

Conclusion: Applications of AI in HPB surgery primarily focus on image analysis and computer vision to address diagnostic and prognostic uncertainties. Virtual 3D and augmented reality models to support complex HPB interventions are also under active development and likely to be used in surgical planning and education. In addition, natural language processing may be helpful in the annotation and phenotyping of disease, leading to new scientific insights.

Keywords

Artificial Intelligence / big data / surgery / liver / pancreas / biliary

References

[1] McCarthy J. What is artificial intelligence? Available from: https://www.diochnos.com/about/McCarthyWhatisAI.pdf [Last accessed on 23 Mar 2023]

[2] Hashimoto DA,Rus D. Artificial intelligence in surgery: promises and perils. Available from: https://journals.lww.com/annalsofsurgery/Abstract/2018/07000/Artificial_Intelligence_in_Surgery__Promises_and.13.aspx [Last accessed on 23 Mar 2023]

[3] Gumbs AA,Karcz K.White paper: definitions of artificial intelligence and autonomous actions in clinical surgery.Art Int Surg2022;2:93-100

[4] Gumbs AA,d’Allemagne B.What is Artificial Intelligence Surgery?.Art Int Surg2021;1:1-10

[5] Elyan E,Johnston P.Computer vision and machine learning for medical image analysis: recent advances, challenges, and way forward.Art Int Surg2022:2

[6] Bari H,Dasari BVM.Role of artificial intelligence in hepatobiliary and pancreatic surgery.World J Gastrointest Surg2021;13:7-18 PMCID:PMC7830072

[7] Mauro A, Greco M, Grimaldi M. What is big data?.Am J Phys2015;1644:97-104

[8] Vedula SS.Surgical data science: The new knowledge domain.Innov Surg Sci2017;2:109-21 PMCID:PMC5602563

[9] NHS England. 2022/23 priorities and operational planning guidance. Available from: https://www.england.nhs.uk/wp-content/uploads/2022/02/20211223-B1160-2022-23-priorities-and-operational-planning-guidance-v3.2.pdf [Last accessed on 23 Mar 2023]

[10] Tricco AC,Zarin W.PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation.Ann Intern Med2018;169:467-73

[11] Knight SR,Maimbo M,Fairfield CJ.Systematic review of the use of big data to improve surgery in low- and middle-income countries.Br J Surg2019;106:e62-72 PMCID:PMC6590290

[12] Covidence. Veritas health innovation, Melbourne, Australia. Available from: https://www.covidence.org/ [Last accessed on 23 Mar 2023]

[13] Săftoiu A,Gorunescu F.Efficacy of an artificial neural network-based approach to endoscopic ultrasound elastography in diagnosis of focal pancreatic masses.Clin Gastroenterol Hepatol2012;10:84-90.e1

[14] Wu K,Ding M.Deep learning based classification of focal liver lesions with contrast-enhanced ultrasound.Optik2014;125:4057-63

[15] Gatos I,Karamesini M,Kagadis G.Development of a support vector machine - based image analysis system for focal liver lesions classification in magnetic resonance images.J Phys Conf Ser2015;633:012116

[16] Roch AM,Krishnan A.Automated pancreatic cyst screening using natural language processing: a new tool in the early detection of pancreatic cancer.HPB2015;17:447-53 PMCID:PMC4402056

[17] Sada Y,Richardson P,Davila J.Validation of case finding algorithms for hepatocellular cancer from administrative data and electronic health records using natural language processing.Med Care2016;54:e9-14 PMCID:PMC3875602

[18] Kondo S,Nishida M.Computer-aided diagnosis of focal liver lesions using contrast-enhanced ultrasonography with perflubutane microbubbles.IEEE Trans Med Imaging2017;36:1427-37

[19] Yang H,Cai XY.From big data to diagnosis and prognosis: gene expression signatures in liver hepatocellular carcinoma.PeerJ2017;5:e3089 PMCID:PMC5354077

[20] Kuwahara T,Mizuno N.Usefulness of deep learning analysis for the diagnosis of malignancy in intraductal papillary mucinous neoplasms of the pancreas.Clin Transl Gastroenterol2019;10:1-8 PMCID:PMC6602761

[21] Shen X,Yang P. Non-invasive diagnosis model for pancreatic cystic tumors based on machine learning-radiomics using contrast-enhanced CT. Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3294088 [Last accessed on 27 Mar 2023]

[22] Xu L,Liang W.A radiomics approach based on support vector machine using MR images for preoperative lymph node status evaluation in intrahepatic cholangiocarcinoma.Theranostics2019;9:5374-85 PMCID:PMC6691572

[23] Brown AD.Natural language processing of radiology reports in patients with hepatocellular carcinoma to predict radiology resource utilization.J Am Coll Radiol2019;16:840-4

[24] Watson MD,Passeri MJ.Use of artificial intelligence deep learning to determine the malignant potential of pancreatic cystic neoplasms with preoperative computed tomography imaging.Am Surg2021;87:602-7

[25] Liu H,Zhang Z.A natural language processing pipeline of chinese free-text radiology reports for liver cancer diagnosis.IEEE Access2020;8:159110-9

[26] Mao B,Duan S,Tao Y.Preoperative classification of primary and metastatic liver cancer via machine learning-based ultrasound radiomics.Eur Radiol2021;31:4576-86

[27] Jang SI,Kim EJ.Diagnostic performance of endoscopic ultrasound-artificial intelligence using deep learning analysis of gallbladder polypoid lesions.J Gastroenterol Hepatol2021;36:3548-55

[28] Li D,Shen Y,Lv H.Artificial intelligence-assisted visual sensing technology under duodenoscopy of gallbladder stones.J Sensors2021;2021:1-13

[29] Kim T,Choi JH,Lee S.Gallbladder polyp classification in ultrasound images using an ensemble convolutional neural network model.J Clin Med2021;10:3585 PMCID:PMC8396835

[30] Yamashita R,Cheung PY.Automated identification and measurement extraction of pancreatic cystic lesions from free-text radiology reports using natural language processing.Radiol Artif Intell2022;4:e210092 PMCID:PMC8980879

[31] Chong H,Zhang Y,Sheng R.Radiomics on gadoxetate disodium-enhanced mri: non-invasively identifying glypican 3-positive hepatocellular carcinoma and postoperative recurrence.Acad Radiol2023;30:49-63

[32] Liu Y,Sun L,Inomoto C.Subtyping of hepatocellular adenoma: a machine learning-based approach.Virchows Arch2022;481:49-61

[33] Schuessler M,Al-Rashid F.Diagnostic accuracy of coronary computed tomography angiography-derived fractional flow reserve (CT-FFR) in patients before liver transplantation using CT-FFR machine learning algorithm.Eur Radiol2022;32:8761-8 PMCID:PMC9705488

[34] Chang Y,Chi L,Li Q.Adoption of combined detection technology of tumor markers via deep learning algorithm in diagnosis and prognosis of gallbladder carcinoma.J Supercomput2022;78:3955-75

[35] Kooragayala K,Kalola A.Utilization of natural language processing software to identify worrisome pancreatic lesions.Ann Surg Oncol2022;29:8513-9

[36] Singal AG,Elmunzer BJ.Machine learning algorithms outperform conventional regression models in predicting development of hepatocellular carcinoma.Am J Gastroenterol2013;108:1723-30 PMCID:PMC4610387

[37] Banerjee S,Kim HJ.A computed tomography radiogenomic biomarker predicts microvascular invasion and clinical outcomes in hepatocellular carcinoma.Hepatology2015;62:792-800 PMCID:PMC4654334

[38] Walczak S.An evaluation of artificial neural networks in predicting pancreatic cancer survival.J Gastrointest Surg2017;21:1606-12

[39] Zhou Y,Huang Y.CT-based radiomics signature: a potential biomarker for preoperative prediction of early recurrence in hepatocellular carcinoma.Abdom Radiol2017;42:1695-704

[40] Zheng BH,Zhang ZZ.Radiomics score: a potential prognostic imaging feature for postoperative survival of solitary HCC patients.BMC Cancer2018;18:1148 PMCID:PMC6249916

[41] Ivanics T,Patel MS.The toronto postliver transplantation hepatocellular carcinoma recurrence calculator: a machine learning approach.Liver Transpl2022;28:593-602

[42] Sala Elarre P,Yu KH.Use of machine-learning algorithms in intensified preoperative therapy of pancreatic cancer to predict individual risk of relapse.Cancers2019;11:606 PMCID:PMC6562932

[43] Marinelli B,Martini M.Combination of active transfer learning and natural language processing to improve liver volumetry using surrogate metrics with deep learning.Radiol Artif Intell2019;1:e180019 PMCID:PMC8017413

[44] Nasief H,Schott D.A machine learning based delta-radiomics process for early prediction of treatment response of pancreatic cancer.NPJ Precis Oncol2019;3:25 PMCID:PMC6778189

[45] Shan QY,Feng ST.CT-based peritumoral radiomics signatures to predict early recurrence in hepatocellular carcinoma after curative tumor resection or ablation.Cancer Imaging2019;19:11

[46] Chen Y,Mo Y.Prediction of post-hepatectomy liver failure in patients with hepatocellular carcinoma based on radiomics using Gd-EOB-DTPA-enhanced MRI: the liver failure model.Front Oncol2021;11:605296 PMCID:PMC7987905

[47] Han IW,Ryu Y.Risk prediction platform for pancreatic fistula after pancreatoduodenectomy using artificial intelligence.World J Gastroenterol2020;26:4453-64 PMCID:PMC7438201

[48] Kambakamba P,Herrera P.Machine learning based texture analysis predicts postoperative pancreatic fistula in preoperative non-contrast enhanced computed tomography.HPB2020;22:S384

[49] Merath K,Mehta R.Use of machine learning for prediction of patient risk of postoperative complications after liver, pancreatic, and colorectal surgery.J Gastrointest Surg2020;24:1843-51

[50] Saillard C,Laifa O.Predicting survival after hepatocellular carcinoma resection using deep learning on histological slides.Hepatology2020;72:2000-13

[51] Cesaretti M,Goumard C.Use of artificial intelligence as an innovative method for liver graft macrosteatosis assessment.Liver Transpl2020;26:1224-32

[52] Mai RY,Bai T.Artificial neural network model for preoperative prediction of severe liver failure after hemihepatectomy in patients with hepatocellular carcinoma.Surgery2020;168:643-52

[53] Liu CL,Lee WC,Lin YC.Predicting short-term survival after liver transplantation using machine learning.Sci Rep2020;10:5654 PMCID:PMC7101323

[54] Schoenberg MB,Koch D.A novel machine learning algorithm to predict disease free survival after resection of hepatocellular carcinoma.Ann Transl Med2020;8:434 PMCID:PMC7210189

[55] Szpakowski JL.Outcomes of gallbladder polyps and their association with gallbladder cancer in a 20-year cohort.JAMA Netw Open2020;3:e205143 PMCID:PMC7235691

[56] Capretti G,De Palma C.A machine learning risk model based on preoperative computed tomography scan to predict postoperative outcomes after pancreatoduodenectomy.Updates Surg2022;74:235-43

[57] Sun LY,Cen WJ,Tang WT.A model based on artificial intelligence algorithm for monitoring recurrence of HCC after hepatectomy.Am Surg2021;11:31348211063549

[58] Xie F,Zhou Y.Characterization of patients with advanced chronic pancreatitis using natural language processing of radiology reports.PLoS One2020;15:e0236817 PMCID:PMC7437899

[59] Hayashi K,Takamatsu M.Prediction of recurrence pattern of pancreatic cancer post-pancreatic surgery using histology-based supervised machine learning algorithms: a single-center retrospective study.Ann Surg Oncol2022:4624-34

[60] Li X,Lou J.Preoperative recurrence prediction in pancreatic ductal adenocarcinoma after radical resection using radiomics of diagnostic computed tomography.EClinicalMedicine2022;43:101215 PMCID:PMC8649647

[61] Noh B,Kwon Y.Machine learning-based survival rate prediction of Korean hepatocellular carcinoma patients using multi-center data.BMC Gastroenterol2022;22:85 PMCID:PMC8882306

[62] Morris-Stiff G,Hu B.The natural history of asymptomatic gallstones: a longitudinal study and prediction model.Clin Gastroenterol Hepatol2023;21:319-327.e4

[63] Narayan RR,Yang L.Artificial intelligence for prediction of donor liver allograft steatosis and early post-transplantation graft failure.HPB2022;24:764-71

[64] Cotter G,Poultsides GA.Using machine learning to preoperatively stratify prognosis among patients with gallbladder cancer: a multi-institutional analysis.HPB2022;24:1980-8

[65] Spinczyk D,Rudnicki J.Stereoscopic liver surface reconstruction.Wideochir Inne Tech Maloinwazyjne2012;7:181-7 PMCID:PMC3516990

[66] Okamoto T,Matsumoto M.Utility of augmented reality system in hepatobiliary surgery.J Hepatobiliary Pancreat Sci2013;20:249-53

[67] Fang CH,Fan YF,Xiang N.Outcomes of hepatectomy for hepatolithiasis based on 3-dimensional reconstruction technique.J Am Coll Surg2013;217:280-8

[68] Zein NN,Bishop PD.Three-dimensional print of a liver for preoperative planning in living donor liver transplantation.Liver Transpl2013;19:1304-10

[69] Shahin O,Kleemann M.Ultrasound-based tumor movement compensation during navigated laparoscopic liver interventions.Surg Endosc2014;28:1734-41

[70] Yang X,Choi Y.Development and usability testing of Dr. Liver^TM: a user-centered 3D virtual liver surgery planning system.HFES2014;58:698-702

[71] Fang CH,Wang X.Three-dimensional reconstruction of the peripancreatic vascular system based on computed tomographic angiography images and its clinical application in the surgical management of pancreatic tumors.Pancreas2014;43:389-95

[72] Bégin A,Lapointe R.Accuracy of preoperative automatic measurement of the liver volume by CT-scan combined to a 3D virtual surgical planning software (3DVSP).Surg Endosc2014;28:3408-12

[73] Bliznakova K,Buliev I.Computer aided preoperative evaluation of the residual liver volume using computed tomography images.J Digit Imaging2015;28:231-9 PMCID:PMC4359196

[74] Katić D,Wekerle AL.LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognition.Int J Comput Assist Radiol Surg2015;10:1427-34

[75] Song Y,Thompson S.Locally rigid, vessel-based registration for laparoscopic liver surgery.Int J Comput Assist Radiol Surg2015;10:1951-61 PMCID:PMC4642598

[76] Wang G,Xie H,Gu L.A homotopy-based sparse representation for fast and accurate shape prior modeling in liver surgical planning.Med Image Anal2015;19:176-86

[77] Fang CH,Yang J.Impact of three-dimensional reconstruction technique in the operation planning of centrally located hepatocellular carcinoma.J Am Coll Surg2015;220:28-37

[78] Zhang J,Guo XC.Application of three-dimensional visualization technique in preoperative planning of progressive hilar cholangiocarcinoma.Am J Transl Res2018;10:1730-5 PMCID:PMC6038071

[79] Okuda Y,Seo S.Usefulness of operative planning based on 3-dimensional CT cholangiography for biliary malignancies.Surgery2015;158:1261-71

[80] Okamoto T,Yasuda J,Suzuki N.Navigation surgery using an augmented reality for pancreatectomy.Dig Surg2015;32:117-23

[81] Fortmeier D,Schröder J.A virtual reality system for PTCD simulation using direct visuo-haptic rendering of partially segmented image data.IEEE J Biomed Health Inform2016;20:355-66

[82] Fusaglia M,Schwalbe M.A clinically applicable laser-based image-guided system for laparoscopic liver procedures.Int J Comput Assist Radiol Surg2016;11:1499-513

[83] Ntourakis D,Soler L,Mutter D.Augmented reality guidance for the resection of missing colorectal liver metastases: an initial experience.World J Surg2016;40:419-26

[84] Mastmeyer A,Handels H.Evaluation of direct haptic 4D volume rendering of partially segmented data for liver puncture simulation.Sci Rep2017;7:671 PMCID:PMC5429645

[85] Sauer IM,Tang P.Mixed reality in visceral surgery: development of a suitable workflow and evaluation of intraoperative use-cases.Ann Surg2017;266:706-12

[86] Cai W,Hu H,Fang C.Postoperative liver volume was accurately predicted by a medical image three dimensional visualization system in hepatectomy for liver cancer.Surg Oncol2017;26:188-94

[87] Miyamoto R,Nakayama K.Three-dimensional simulation of pancreatic surgery showing the size and location of the main pancreatic duct.Surg Today2017;47:357-64

[88] Hu M,Cai W.The safety and feasibility of three-dimensional visualization technology assisted right posterior lobe allied with part of V and VIII sectionectomy for right hepatic malignancy therapy.J Laparoendosc Adv Surg Tech A2018;28:586-94

[89] Mise Y,Satou S.How has virtual hepatectomy changed the practice of liver surgery?.Ann Surg2018;268:127-33

[90] Mascagni P,Urade T.Formalizing video documentation of the critical view of safety in laparoscopic cholecystectomy: a step towards artificial intelligence assistance to improve surgical safety.Surg Endosc2020;34:2709-14

[91] Teatini A,Aghayan DL.The effect of intraoperative imaging on surgical navigation for laparoscopic liver resection surgery.Sci Rep2019;9:18687 PMCID:PMC6904553

[92] Ho H,Bartlett A.An in silico pipeline for subject-specific hemodynamics analysis in liver surgery planning.Comput Methods Biomech Biomed Engin2020;23:138-42

[93] Prevost GA,Paolucci I.Efficiency, accuracy and clinical applicability of a new image-guided surgery system in 3D laparoscopic liver surgery.J Gastrointest Surg2020;24:2251-8

[94] Sandal B,Salihoglu Z.Fuzzy logic preanesthetic risk evaluation of laparoscopic cholecystectomy operations.Am Surg2023;89:414-23

[95] Cervantes-sanchez F,Köhler H.Automatic tissue segmentation of hyperspectral images in liver and head neck surgeries using machine learning.Art Int Surg2021;1:22-37

[96] Tokuyasu T,Matsunobu Y.Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomy.Surg Endosc2021;35:1651-8 PMCID:PMC7940266

[97] Guzmán-García C,Sánchez-González P,Gómez EJ.Speech-based surgical phase recognition for non-intrusive surgical skills' assessment in educational contexts.Sensors2021;21:1330 PMCID:PMC7918578

[98] Imler TD,Imperiale TF.Provider-specific quality measurement for ERCP using natural language processing.Gastrointest Endosc2018;87:164-173.e2 PMCID:PMC5670027

[99] Ruzzenente A,Poletto E.A machine learning analysis of difficulty scoring systems for laparoscopic liver surgery.Surg Endosc2022;36:8869-80 PMCID:PMC9652257

[100] Mascagni P,Laracca GG.Multicentric validation of EndoDigest: a computer vision platform for video documentation of the critical view of safety in laparoscopic cholecystectomy.Surg Endosc2022;36:8379-86

[101] Mascagni P,Alapatt D.Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning.Ann Surg2022;275:955-61

[102] Tranter-entwistle I,Connor S.Operative difficulty in laparoscopic cholecystectomy: considering the role of machine learning platforms in clinical practice.Art Int Surg2022;2:46-56

[103] Liu R,Wang Z.Artificial intelligence in laparoscopic cholecystectomy: does computer vision outperform human vision?.Art Int Surg2022;2:80-92

[104] Ugail H,Elmahmudi A,Thomson B.The use of pre-trained deep learning models for the photographic assessment of donor livers for transplantation.Art Int Surg2022;2:101-19

[105] Mojtahed A,Connell J.Repeatability and reproducibility of deep-learning-based liver volume and Couinaud segment volume measurement tool.Abdom Radiol2022;47:143-51 PMCID:PMC8776724

[106] Han X,Wang S.Automated segmentation of liver segment on portal venous phase MR images using a 3D convolutional neural network.Insights Imaging2022;13:26 PMCID:PMC8873293

[107] Ward TM,Ban Y,Meireles OR.Artificial intelligence prediction of cholecystectomy operative course from automated identification of gallbladder inflammation.Surg Endosc2022;36:6832-40

[108] Madani A,Altieri MS.Artificial intelligence for intraoperative guidance: using semantic segmentation to identify surgical anatomy during laparoscopic cholecystectomy.Ann Surg2022;276:363-9 PMCID:PMC8186165

[109] Loukas C,Schizas D.Multiple instance convolutional neural network for gallbladder assessment from laparoscopic images.Int J Med Robot2022;18:e2445

[110] Golany T,Freedman D.Artificial intelligence for phase recognition in complex laparoscopic cholecystectomy.Surg Endosc2022;36:9215-23 PMCID:PMC9652206

[111] 2019 Diseases and Injuries Collaborators. Global burden of 369 diseases and injuries in 204 countries and territories, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019.Lancet2020;396:1204-22 PMCID:PMC7567026

[112] Rajpurkar P,Banerjee O.AI in health and medicine.Nat Med2022;28:31-8

[113] Reddy S.Explainability and artificial intelligence in medicine.Lancet Digit Health2022;4:e214-5

[114] Ghassemi M,Beam AL.The false hope of current approaches to explainable artificial intelligence in health care.Lancet Digit Health2021;3:e745-50

[115] Topol EJ.High-performance medicine: the convergence of human and artificial intelligence.Nat Med2019;25:44-56

[116] He J,Xu J,Zhou X.The practical implementation of artificial intelligence technologies in medicine.Nat Med2019;25:30-6 PMCID:PMC6995276

[117] Rivera SC,Chan AW.Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension.Lancet Digit Health2020;2:e549-60

[118] Collins GS,Navarro CLA.Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence.BMJ Open2021;11:e048008 PMCID:PMC8273461