Deep learning in real-time image-guided surgery: a systematic review of applications, methodologies, and clinical relevance

Abstract

Aim: Real-time image guidance using deep learning is being increasingly used in surgery. This systematic review aims to characterize intraoperative systems, mapping applications, performance and latency, validation practices, and the reported effects on workflow and patient-relevant outcomes.

Methods: A systematic review was conducted on PubMed, Embase, Scopus, ScienceDirect, IEEE Xplore, Google Scholar, and Directory of Open Access Journals from December 31, 2024. Eligible English-language, peer-reviewed diagnostic accuracy, cohort, quasi-experimental, or randomized studies (2017-2024) evaluated the learning for real-time intraoperative guidance. Two reviewers screened, applied the Joanna Briggs Institute checklists, and extracted the design, modality, architecture, training, validation, performance, and latency. Heterogeneity precluded the meta-analysis.

Results: Twenty-seven studies spanning laparoscopic, neurosurgical, breast, colorectal, cardiac, and other workflows met the criteria. The modalities included red-green-blue laparoscopy or endoscopy, ultrasound, optical coherence tomography, cone-beam computed tomography, and stimulated Raman histology. The architectures were mainly convolutional neural networks with frequent transfer learning. Reported performance was high, with classification accuracy commonly 90%-97% and segmentation Dice or intersection over union up to 0.95 at operating-room-compatible speeds of about 20-300 frames per second or sub-second per-frame latency; volumetric pipelines sometimes required up to 1 min. Several systems demonstrated intraoperative feasibility and high surgeon acceptance, yet fewer than one quarter reported external validation and only a small subset linked outputs to patient-important outcomes.

Conclusion: Deep-learning systems for real-time image guidance exhibit strong technical performance and emerging workflow benefits. Priorities include multicenter prospective evaluations, standardized reporting of latency and external validation, rigorous human factors assessment, and open benchmarking to demonstrate generalizability and patient impact.

Keywords

Deep learning / image-guided surgery / intraoperative imaging / real-time guidance / convolutional neural networks / systematic review

References

[1] Meara JG,Hagander L.Global Surgery 2030: evidence and solutions for achieving health, welfare, and economic development.Lancet2015;386:569-624

[2] Nepogodiev D,Biccard B,Bhangu A.National Institute for Health Research Global Health Research Unit on Global SurgeryGlobal burden of postoperative death.Lancet2019;393:401

[3] Keil H,Swartman B.Intraoperative revision rates due to three-dimensional imaging in orthopedic trauma surgery: results of a case series of 4721 patients.Eur J Trauma Emerg Surg2023;49:373-81 PMCID:PMC9925545

[4] Wang X,Zhou B,Liang Y.Integrating mixed reality, augmented reality, and artificial intelligence in complex liver surgeries: enhancing precision, safety, and outcomes.ILIVER2025;4:100167 PMCID:PMC12209476

[5] Singla N,Srivastava V.Automated assessment of breast cancer margin in optical coherence tomography images via pretrained convolutional neural network.J Biophotonics2019;12:e201800255

[6] Hollon TC,Adapa AR.Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks.Nat Med2020;26:52-8 PMCID:PMC6960329

[7] Li Y,Li R.Deep learning for surgical workflow analysis: a survey of progresses, limitations, and trends.Artif Intell Rev2024;57:10929

[8] Kitaguchi D,Matsuzaki H.Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach.Surg Endosc2020;34:4924-31

[9] Aboy M,Vayena E.Navigating the EU AI Act: implications for regulated digital medical products.NPJ Digit Med2024;7:237 PMCID:PMC11379845

[10] Magalhães R,Terroso D.Mixed reality in the operating room: a systematic review.J Med Syst2024;48:76 PMCID:PMC11327191

[11] Nair M,Larsson I.A comprehensive overview of barriers and strategies for AI implementation in healthcare: mixed-method design.PLoS One2024;19:e0305949 PMCID:PMC11315296

[12] Ennab M.Enhancing interpretability and accuracy of AI models in healthcare: a comprehensive review on challenges and future directions.Front Robot AI2024;11:1444763 PMCID:PMC11638409

[13] Salahuddin Z,Chatterjee A.Transparency of deep neural networks for medical image analysis: a review of interpretability methods.Comput Biol Med2022;140:105111

[14] U. S. Food & Drug Administration. FDA Roundup: September 17, 2024. Available from: https://www.fda.gov/news-events/press-announcements/fda-roundup-september-17-2024?utm_source=chatgpt.com [accessed 16 December 2025].

[15] Schneider C,Totz J.Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: a clinical feasibility study.Surg Endosc2020;34:4702-11 PMCID:PMC7524854

[16] Koetzier LR,Mastrodicasa D.Generating synthetic data for medical imaging.Radiology2024;312:e232471 PMCID:PMC11444329

[17] PRISMA 2020. PRISMA statement. Available from: https://www.prisma-statement.org/ [accessed 16 December 2025].

[18] Sasaki K,Kobayashi S.Automated surgical workflow identification by artificial intelligence in laparoscopic hepatectomy: experimental research.Int J Surg2022;105:106856

[19] Padovan E,Tanzi L.A deep learning framework for real-time 3D model registration in robot-assisted laparoscopic surgery.Int J Med Robot2022;18:e2387 PMCID:PMC9286374

[20] Boonkong A,Kaewfoongrungsi P,Hormdee D.Applying deep learning for occluded uterus and fallopian tube detection for laparoscopic tubal sterilization.IEEE Access2024;12:183182-94

[21] Liu Z,Li H.Automated deep neural network-based identification, localization, and tracking of cardiac structures for ultrasound-guided interventional surgery.J Thorac Dis2023;15:2129-40 PMCID:PMC10183525

[22] Aubreville M,Oetter N.Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning.Sci Rep2017;7:11979 PMCID:PMC5607286

[23] Nwoye CI,Sharma S.CholecTriplet2022: Show me a tool and tell me the triplet - an endoscopic vision challenge for surgical action triplet detection.Med Image Anal2023;89:102888

[24] Zhang X,Zbijewski WB.Combining physics-based models with deep learning image synthesis and uncertainty in intraoperative cone-beam CT of the brain.Med Phys2023;50:2607-24 PMCID:PMC10175241

[25] Caballas KG,Libatique NJC.Development of a visual guidance system for laparoscopic surgical palpation using computer vision. 2020 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES); 2021 Mar 1-3; Langkawi Island, Malaysia. New York: IEEE; 2021. pp. 88-93.

[26] Smithmaitrie P,Sae-Lim W,Jearanai S.Development of deep learning framework for anatomical landmark detection and guided dissection line during laparoscopic cholecystectomy.Heliyon2024;10:e25210 PMCID:PMC10847946

[27] Török P.Digital image analysis with fully connected convolutional neural network to facilitate hysteroscopic fibroid resection.Gynecol Obstet Invest2018;83:615-9

[28] Lin J,Qi J.Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks.Med Image Anal2018;48:162-76

[29] Blokker M,Wesseling P,Veta M.Fast intraoperative histology-based diagnosis of gliomas with third harmonic generation microscopy and deep learning.Sci Rep2022;12:11334 PMCID:PMC9256596

[30] Mojahed D,Chang P.Fully automated postlumpectomy breast margin assessment utilizing convolutional neural network based optical coherence tomography image classification method.Acad Radiol2020;27:e81-6 PMCID:PMC7456393

[31] Tai Y,Huang X,Jan MA.Intelligent intraoperative haptic-AR navigation for COVID-19 lung biopsy using deep hybrid model.IEEE Trans Industr Inform2021;17:6519-27 PMCID:PMC8545008

[32] Jalal NA,Docherty PD.Laparoscopic video analysis using temporal, attention, and multi-feature fusion based-approaches.Sensors2023;23:1958 PMCID:PMC9964851

[33] Mao Z,Islam M.PitSurgRT: real-time localization of critical anatomical structures in endoscopic pituitary surgery.Int J Comput Assist Radiol Surg2024;19:1053-60 PMCID:PMC11178578

[34] Zeng Y,Chapman WC Jr.Real-time colorectal cancer diagnosis using PR-OCT with deep learning.Theranostics2020;10:2587-96 PMCID:PMC7052898

[35] Tanzi L,Porpiglia F.Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance.Int J Comput Assist Radiol Surg2021;16:1435-45 PMCID:PMC8354939

[36] Podlasek J,Podlasek R,Filip R.Real-time deep learning-based colorectal polyp localization on clinical video footage achievable with a wide array of hardware configurations.Endosc Int Open2021;9:E741-8 PMCID:PMC8062241

[37] Sato K,Matsuzaki H.Real-time detection of the recurrent laryngeal nerve in thoracoscopic esophagectomy using artificial intelligence.Surg Endosc2022;36:5531-9

[38] Canalini L,Miller D.Segmentation-based registration of ultrasound volumes for glioma resection in image-guided neurosurgery.Int J Comput Assist Radiol Surg2019;14:1697-713 PMCID:PMC6797669

[39] Mekki L,Vijayan RC.Surgical navigation for guidewire placement from intraoperative fluoroscopy in orthopedic surgery.Phys Med Biol2023;68:215001

[40] Geldof F,Jong LS,Ruers TJM.Tumor segmentation in colorectal ultrasound images using an ensemble transfer learning model: towards intra-operative margin assessment.Diagnostics2023;13:3595 PMCID:PMC10706767

[41] Kenig N,Muntaner Vives A.Artificial intelligence in surgery: a systematic review of use and validation.J Clin Med2024;13:7108 PMCID:PMC11642125

[42] Yang HY,Yoon J.Deep learning-based surgical phase recognition in laparoscopic cholecystectomy.Ann Hepatobiliary Pancreat Surg2024;28:466-73 PMCID:PMC11599821

[43] Kostiuchik G,Mayer B,Preim B.Surgical phase and instrument recognition: how to identify appropriate dataset splits.Int J Comput Assist Radiol Surg2024;19:699-711 PMCID:PMC10973055

[44] Goyal A,Munoz AE.Artificial intelligence for real-time surgical phase recognition in minimal invasive inguinal hernia repair: a systematic review on behalf of TROGSS - the robotic global surgical society.Art Int Surg2025;5:450-64

[45] Bellini V,Domenichetti T,Allai S.Artificial intelligence in operating room management.J Med Syst2024;48:19 PMCID:PMC10867065

[46] Hollon TC.An automated tissue-to-diagnosis pipeline using intraoperative stimulated Raman histology and deep learning.Mol Cell Oncol2020;7:1736742 PMCID:PMC7199763

[47] Duan Y,Zhang X.Diagnostic accuracy of optical coherence tomography for margin assessment in breast-conserving surgery: a systematic review and meta-analysis.Photodiagnosis Photodyn Ther2023;43:103718

[48] Fan S,Meng Z,Luo Y.Comparing the diagnostic efficacy of optical coherence tomography and frozen section for margin assessment in breast-conserving surgery: a meta-analysis.J Clin Pathol2024;77:517-27

[49] Ramalhinho J,Gurusamy K,Clarkson MJ.Assessing augmented reality displays in laparoscopic liver surgery - a clinical experience.Surg Endosc2025;39:5863-71 PMCID:PMC12408658

[50] Roman J,Němec M.Augmented and mixed reality in liver surgery: a comprehensive narrative review of novel clinical implications on cohort studies.Rev Assoc Med Bras2025;71:e20250315 PMCID:PMC12245099

[51] Raissi-dehkordi N,Xu B.Contemporary applications of artificial intelligence and machine learning in echocardiography.npj Cardiovasc Health2025;2:64

[52] Hirata Y.AI in echocardiography: state-of-the-art automated measurement techniques and clinical applications.JMA J2025;8:141-50 PMCID:PMC11799715

[53] Wagner M,Kisilenko A.Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark.Med Image Anal2023;86:102770

[54] Lavanchy JL,Dall’Alba D.Challenges in multi-centric generalization: phase and step recognition in Roux-en-Y gastric bypass surgery.Int J Comput Assist Radiol Surg2024;19:2249-57 PMCID:PMC11541311

[55] Abdelwanis M,Tammam MMS.Exploring the risks of automation bias in healthcare artificial intelligence applications: a Bowtie analysis.Journal of Safety Science and Resilience2024;5:460-9

[56] Tun HM,Naing L.Trust in artificial intelligence-based clinical decision support systems among health care workers: systematic review.J Med Internet Res2025;27:e69678 PMCID:PMC12440830

[57] Sadeghi Z,Cifci MA.A review of Explainable Artificial Intelligence in healthcare.Comput Electr Eng2024;118:109370

[58] GOV.UK. Equity in medical devices: independent review - final report. Available from: https://www.gov.uk/government/publications/equity-in-medical-devices-independent-review-final-report [accessed 16 December 2025].

[59] European Commission. MDCG 2025-6 - FAQ on interplay between the medical devices regulation & in vitro diagnostic medical devices regulation and the artificial intelligence act (June 2025). Available from: https://health.ec.europa.eu/latest-updates/mdcg-2025-6-faq-interplay-between-medical-devices-regulation-vitro-diagnostic-medical-devices-2025-06-19_en [accessed 16 December 2025].

[60] US Food and Drug Administration. Marketing submission recommendations for a predetermined change control plan for artificial intelligence-enabled device software functions. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence [accessed 16 December 2025].

[61] US Food and Drug Administration. Good machine learning practice for medical device development: guiding principles. Available from: https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles [accessed 16 December 2025].

[62] International Medical Device Regulations Forum. Good machine learning practice for medical device development: guiding principles. AUTHORING GROUP: Artificial Intelligence/Machine Learning-enabled Working Group, 2025. Available from: https://www.imdrf.org/sites/default/files/2025-01/IMDRF_AIML%20WG_GMLP_N88%20Final_0.pdf [accessed 16 December 2025].