Automatic tissue segmentation of hyperspectral images in liver and head neck surgeries using machine learning

Abstract

Aim: Proper identification in real time of different types of tissues during intraoperative procedures represents a vital and challenging task. This paper addresses tissue segmentation in two different medical applications using hyperspectral imaging (HSI) and machine learning in two main steps.

Methods: The first step consists of data preprocessing designed to overcome the most common problems linked with HSI, involving inter- and intra-patient variability of the tissue spectra and the high dimensionality of the spectra. The preprocessing step involves outlier removal, spectral smoothing, normalization, and dimensionality reduction using principal component analysis applied in the spectral domain of HSI data. In the spatial domain, multiple levels of analysis are performed using Gaussian filters. The second step consists of tissue segmentation using an optimized machine learning model. The most suitable model was selected under statistical comparison of seven machine learning models involving three different levels of spatial analysis.

Results: According to the experimental results, the U-Net achieves the highest precision (0.908) for detection of liver, bile duct, artery, and portal vein tissues in a set of 18 HSI data, while the logistic regression with the elasticnet regularization combined with multiscale spatial analysis obtains the highest F1-score (0.673) and segmentation accuracy (0.803) for thyroid and parathyroid glands segmentation in a set of 21 HSI data.

Conclusion: In addition to the computational experiments, combining machine learning with HSI represents a promising approach to perform image-guided surgery.

Keywords

Hyperspectral imaging / tissue segmentation / machine learning / deep learning

References

[1] Fabelo H,Ortega S.Deep learning-based framework for in vivo identification of glioblastoma tumor using hyperspectral images of human brain..Sensors (Basel)2019;19:920 PMCID:PMC6412736

[2] Holmer A,Marotz J.Oxygenation and perfusion monitoring with a hyperspectral camera system for chemical based tissue analysis of skin and organs..Physiol Meas2016;37:2064-78

[3] Kulcke A,Wahl P,Wild T.A compact hyperspectral camera for measurement of perfusion parameters in medicine..Biomed Tech (Berl)2018;63:519-27

[4] Ortega S,Iakovidis DK,Callico GM.Use of hyperspectral/multispectral imaging in gastroenterology. Shedding some-different-light into the dark..J Clin Med2019;8:36 PMCID:PMC6352071

[5] Baltussen EJM,Brouwer de Koning SG.Hyperspectral imaging for tissue classification, a way toward smart laparoscopic colorectal surgery..J Biomed Opt2019;24:1-9 PMCID:PMC6985687

[6] Nouri D,Treuillet S.Hyperspectral interventional imaging for enhanced tissue visualization and discrimination combining band selection methods..Int J Comput Assist Radiol Surg2016;11:2185-97

[7] Shapey J,Nabavi E.Intraoperative multispectral and hyperspectral label-free imaging: a systematic review of in vivo clinical studies..J Biophotonics2019;12:e201800455 PMCID:PMC6736677

[8] Zhang Y,Zhu X.Explainable liver tumor delineation in surgical specimens using hyperspectral imaging and deep learning..Biomed Opt Express2021;12:4510 PMCID:PMC8367264

[9] Halicek M,Little JV,Fei B.Tumor detection of the thyroid and salivary glands using hyperspectral imaging and deep learning..Biomed Opt Express2020;11:1383-400 PMCID:PMC7075628

[10] Halicek M,Ortega S,Fei B.In-vivo and ex-vivo tissue analysis through hyperspectral imaging techniques: revealing the invisible features of cancer..Cancers (Basel)2019;11:756 PMCID:PMC6627361

[11] Schols RM,Stassen LP.Differentiation between nerve and adipose tissue using wide-band (350-1,830 nm) in vivo diffuse reflectance spectroscopy..Lasers Surg Med2014;46:538-45

[12] Cooney GS,Diana M,Chalopin C.Comparison of spectral characteristics in human and pig biliary system with hyperspectral imaging (HSI)..Current Directions in Biomedical Engineering2020;6:20200012

[13] Maktabi M,Ivanova M.Classification of hyperspectral endocrine tissue images using support vector machines..Int J Med Robot2020;16:1-10

[14] Schols RM,Wieringa FP,Stassen LP.Towards automated spectroscopic tissue classification in thyroid and parathyroid surgery..Int J Med Robot2017;13:e1748

[15] Signoroni A,Baronio A.Deep learning meets hyperspectral image analysis: a multidisciplinary review..J Imaging2019;5:52

[16] Savitzky A.Smoothing and differentiation of data by simplified least squares procedures..Anal Chem1964;36:1627-39

[17] Chen Y,Li C,Ghamisi P.Deep feature extraction and classification of hyperspectral images based on convolutional neural networks..IEEE Trans Geosci Remote Sensing2016;54:6232-51

[18] Ronneberger O,Brox T.Navab N,Wells WM.U-Net: convolutional networks for biomedical image segmentation..Medical image computing and computer-assisted intervention - MICCAI 2015.2015;ChamSpringer International Publishing234-41

[19] Bokhorst JM,van Zwam P,van der Laak J.Cardoso MJ,Glocker B,Oguz I.Learning from sparsely annotated data for semantic segmentation in histopathology images..2019;London, United KingdomPMLR84-91

[20] Pedregosa F,Gramfort A.Scikit-learn: machine learning in python..J Mach Learn Res2011;12:2825-30

[21] Paszke A,Massa F.Wallach H,Beygelzimer A,Fox E.PyTorch: an imperative style, high-performance deep learning library..Advances in Neural Information Processing Systems 32.2019;Curran Associates, Inc.8026-37

[22] Ghamisi P,Chen Y,Plaza AJ.Advanced spectral classifiers for hyperspectral images: a review..IEEE Geosci Remote Sens Mag2017;5:8-32