Deep learning-driven catheter tracking from bi-plane X-ray fluoroscopy of 3D printed heart phantoms

Abstract

Minimally invasive surgery (MIS) has changed not only the performance of specific operations but also the more effective strategic approach to all surgeries. Expansion of MIS to more complex surgeries demands further development of new technologies, including robotic surgical systems, navigation, guidance, visualizations, dexterity enhancement, and 3D printing technology. In the cardiovascular domain, 3D printed modeling can play a crucial role in providing improved visualization of the anatomical details and guide precision operations as well as functional evaluation of various congenital and congestive heart conditions. In this work, we propose a novel deep learning-driven tracking method for providing quantitative 3D tracking of mock cardiac interventions on custom-designed 3D printed heart phantoms. In this study, the position of the tip of a catheter is tracked from bi-plane fluoroscopic images. The continuous positioning of the catheter relative to the 3D printed model was co-registered in a single coordinate system using external fiducial markers embedded into the model. Our proposed method has the potential to provide quantitative analysis for training exercises of percutaneous procedures guided by bi-plane fluoroscopy.

Keywords

Catheter tracking / image guidance / deep learning / 3D printing / minimally invasive surgery / 3D trajectory / percutaneous interventions / patient-specific

References

[1] Niu G,Zhang F,Fu Y.Improved surgical instruments without coupled motion used in minimally invasive surgery.Int J Med Robot2018;14:e1942

[2] Bello B,Allaix ME.Impact of minimally invasive surgery on the treatment of benign esophageal disorders.World J Gastroenterol2012;18:6764-70 PMCID:PMC3520165

[3] Gjeraa K,Konge L,Østergaard D.Non-technical skills in minimally invasive surgery teams: a systematic review.Surg Endosc2016;30:5185-99

[4] Eswara JR.Minimally invasive techniques in urology.Surg Oncol Clin N Am2019;28:327-32

[5] Schatz C.Enhanced recovery in a minimally invasive thoracic surgery program.AORN J2015;102:482-92

[6] Devgan L,Durairaj K.Minimally invasive facial cosmetic procedures.Otolaryngol Clin North Am2019;52:443-59

[7] Mack MJ.Minimally invasive cardiac surgery.Surg Endosc2006;20 Suppl 2:S488-92

[8] Gillion JF.Chronic pain and cutaneous sensory changes after inguinal hernia repair: comparison between open and laparoscopic techniques.Hernia1999;3:75-80

[9] Dedemadi G,Karaliotas C,Kouraklis G.Comparison of laparoscopic and open tension-free repair of recurrent inguinal hernias: a prospective randomized study.Surg Endosc2006;20:1099-104

[10] Subramanian VA,Geller CM.Minimally invasive direct coronary artery bypass grafting: two-year clinical experience.Ann Thorac Surg1997;64:1648-55

[11] Stevens JH,Peters WS.Port-access coronary artery bypass grafting: a proposed surgical method.J Thorac Cardiovasc Surg1996;111:567-73

[12] Ota T,Schwartzman D.A highly articulated robotic surgical system for minimally invasive surgery.Ann Thorac Surg2009;87:1253-6 PMCID:PMC2691642

[13] Pyciński B,Bożek P,Dzielicki J. Image navigation in minimally invasive surgery. In: Piętka E, Kawa J, Wieclawek W, editors. Information technologies in biomedicine, volume 4. Cham: Springer International Publishing; 2014. p. 25-34. Available from: https://link.springer.com/chapter/10.1007/978-3-319-06596-0_3.[Lastaccessed on 24 Aug 2023]

[14] Antico M,Wu L.Ultrasound guidance in minimally invasive robotic procedures.Med Image Anal2019;54:149-67

[15] Pisano GP,Edmondson AC.Organizational differences in rates of learning: evidence from the adoption of minimally invasive cardiac surgery.Management Science2001;47:752-68

[16] Moorthy K,Dosis A.Dexterity enhancement with robotic surgery.Surg Endosc2004;18:790-5

[17] Milano EG,Wray J.Current and future applications of 3D printing in congenital cardiology and cardiac surgery.Br J Radiol2019;92:20180389 PMCID:PMC6404827

[18] Min JK,Dunham S. 3D Printing applications in cardiovascular medicine. Cambridge: Academic Press; 2018.

[19] Vukicevic M,Min JK.Cardiac 3D printing and its future directions.JACC Cardiovasc Imaging2017;10:171-84 PMCID:PMC5664227

[20] Biglino G,Binazzi A.Virtual and real bench testing of a new percutaneous valve device: a case study.EuroIntervention2012;8:120-8

[21] Schmauss D,Hagl C.Three-dimensional printing in cardiac surgery and interventional cardiology: a single-centre experience.Eur J Cardiothorac Surg2015;47:1044-52

[22] Ryan JR,Richardson R,Nigro JJ.A novel approach to neonatal management of tetralogy of Fallot, with pulmonary atresia, and multiple aortopulmonary collaterals.JACC Cardiovasc Imaging2015;8:103-4

[23] Chaowu Y,Xin S.Three-dimensional printing as an aid in transcatheter closure of secundum atrial septal defect with rim deficiency: in vitro trial occlusion based on a personalized heart model.Circulation2016;133:e608-10

[24] Sodian R,Markert M.Stereolithographic models for surgical planning in congenital heart surgery.Ann Thorac Surg2007;83:1854-7

[25] Noecker AM,Zhou Q.Development of patient-specific three-dimensional pediatric cardiac models.ASAIO J2006;52:349-53

[26] Vranicar M,Douglas WI,Di Sessa TG.The use of stereolithographic hand held models for evaluation of congenital anomalies of the great arteries.Stud Health Technol Inform2008;132:538-43

[27] Schievano S,Coats L.Percutaneous pulmonary valve implantation based on rapid prototyping of right ventricular outflow tract and pulmonary trunk from MR data.Radiology2007;242:490-7

[28] Vukicevic M,Jaeggli M.Control of respiration-driven retrograde flow in the subdiaphragmatic venous return of the Fontan circulation.ASAIO J2014;60:391-9 PMCID:PMC4225011

[29] Garekar S,Chokhandre M.Clinical application and multidisciplinary assessment of three dimensional printing in double outlet right ventricle with remote ventricular septal defect.World J Pediatr Congenit Heart Surg2016;7:344-50

[30] Deferm S,Vlasselaers D.3D-printing in congenital cardiology: from flatland to spaceland.J Clin Imaging Sci2016;6:8 PMCID:PMC4860452

[31] Kiraly L,Jha NK.Three-dimensional printed prototypes refine the anatomy of post-modified Norwood-1 complex aortic arch obstruction and allow presurgical simulation of the repair.Interact Cardiovasc Thorac Surg2016;22:238-40

[32] Biglino G,Taylor AM.3D Printing Cardiovascular Anatomy: A Single-Centre Experience. In: Shishkovsky IV, editor. New Trends in 3D Printing. IntechOpen; 2016.

[33] Anwar S,Varughese J.3D printing in complex congenital heart disease: across a spectrum of age, pathology, and imaging techniques.JACC Cardiovasc Imaging2017;10:953-6

[34] Olivieri LJ,Loke YH,Kim PC.Three-dimensional printing of intracardiac defects from three-dimensional echocardiographic images: feasibility and relative accuracy.J Am Soc Echocardiogr2015;28:392-7

[35] Jang S,Dhrif H.Development of a hybrid training simulator for structural heart disease interventions.Advanced Intelligent Systems2020;2:2000109

[36] Girshick R,Darrell T. Rich feature hierarchies for accurate object detection and semantic segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition; 2014 Jun 23-28; Columbus, USA. 2014.

[37] Jia Y,Donahue J. 2014. Caffe: convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia (MM '14); New York, USA. 2014.

[38] Sermanet P,Zhang X,Fergus R. Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv. [Preprint.] Febuary 24, 2014 [accessed 2023 August 24]. Available from: https://arxiv.org/abs/1312.6229

[39] Chandan G,Jain H. Real time object detection and tracking using deep learning and OpenCV. 2018 International Conference on Inventive Research in Computing Applications (ICIRCA); 2018 Jul 11-12; Coimbatore, India. 2018.

[40] Rajchl M,Oktay O.DeepCut: object segmentation from bounding box annotations using convolutional neural networks.IEEE Trans Med Imaging2017;36:674-83 PMCID:PMC7115996

[41] Ronneberger O,Brox T. U-Net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A, editors. Medical image computing and computer-assisted intervention - MICCAI 2015. Cham: Springer; 2015. p. 234-41. Available from: https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28.[Lastaccessed on 24 Aug 2023]

[42] Shelhamer E,Darrell T.Fully convolutional networks for semantic segmentation.IEEE Trans Pattern Anal Mach Intell2017;39:640-51

[43] Çiçek Ö,Lienkamp SS,Ronneberger O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin S, Joskowicz L, Sabuncu M, Unal G, Wells W, editors. Medical image computing and computer-assisted intervention - MICCAI 2016. Cham: Springer; 2016. p. 424-32.

[44] Milletari F,Ahmadi S. V-Net: fully convolutional neural networks for volumetric medical image segmentation. 2016 Fourth International Conference on 3D Vision (3DV); 2016 Oct 25-28; Stanford, USA. 2016.

[45] Li X,Qi X,Fu CW.H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes.IEEE Trans Med Imaging2018;37:2663-74

[46] Zhou Z,Tajbakhsh N. UNet++: a nested U-Net architecture for medical image segmentation. In: Stoyanov D, Taylor Z, Carneiro G, Syeda-mahmood T, Martel A, Maier-hein L, Tavares JMR, Bradley A, Papa JP, Belagiannis V, Nascimento JC, Lu Z, Conjeti S, Moradi M, Greenspan H, Madabhushi A, editors. Deep learning in medical image analysis and multimodal learning for clinical decision support. Cham: Springer International Publishing; 2018. p. 3-11.

[47] Zhang J,Xu J,Zhang Y.Mdu-net: multi-scale densely connected u-net for biomedical image segmentation.Health Inf Sci Syst2023;11:13

[48] Jin Q,Pham TD.DUNet: a deformable network for retinal vessel segmentation.Knowledge-Based Systems2019;178:149-62

[49] Jin Q,Sun C,Su R.RA-UNet: a hybrid deep attention-aware network to extract liver and tumor in CT scans.Front Bioeng Biotechnol2020;8:605132 PMCID:PMC7785874

[50] Dolz J,Desrosiers C. Dense multi-path U-Net for ischemic stroke lesion segmentation in multiple image modalities. In: Crimi A, Bakas S, Kuijf H, Keyvan F, Reyes M, van Walsum T, editors. Brainlesion: glioma, multiple sclerosis, stroke and traumatic brain injuries. Cham: Springer; 2018. p. 271-82.

[51] Xiao W,Lin Y.Distinct proteome remodeling of industrial saccharomyces cerevisiae in response to prolonged thermal stress or transient heat shock.J Proteome Res2018;17:1812-25

[52] Isensee F,Klein A. Abstract: nnu-net: self-adapting framework for u-net-based medical image segmentation. Available from: https://link.springer.com/chapter/10.1007/978-3-658-25326-4_7.[Lastaccessed on 24 Aug 2023]

[53] Kingma DP. Adam: a method for stochastic optimization. arXiv. [Preprint.] January 30, 2017 [accessed 2023 August 24]. Available from: https://arxiv.org/abs/1412.6980

[54] Fukumoto Y,Tsuchihashi M,Takeshita A.The incidence and risk factors of cholesterol embolization syndrome, a complication of cardiac catheterization: a prospective study.J Am Coll Cardiol2003;42:211-6

[55] Loffroy R,Cercueil JP.Endovascular therapeutic embolisation: an overview of occluding agents and their effects on embolised tissues.Curr Vasc Pharmacol2009;7:250-63

[56] Dice LR.Measures of the amount of ecologic association between species.Ecology1945;26:297-302

[57] Sra J,Choudhuri I.Identifying the third dimension in 2D fluoroscopy to create 3D cardiac maps.JCI Insight2016;1:e90453 PMCID:PMC5161213

[58] Liu J,Singh G.An augmented reality system for image guidance of transcatheter procedures for structural heart disease.PLoS One2019;14:e0219174 PMCID:PMC6602420