Deep Learning in Medical Ultrasound Analysis: A Review

Abstract

Ultrasound (US) has become one of the most commonly performed imaging modalities in clinical practice. It is a rapidly evolving technology with certain advantages and with unique challenges that include low imaging quality and high variability. From the perspective of image analysis, it is essential to develop advanced automatic US image analysis methods to assist in US diagnosis and/or to make such assessment more objective and accurate. Deep learning has recently emerged as the leading machine learning tool in various research fields, and especially in general imaging analysis and computer vision. Deep learning also shows huge potential for various automatic US image analysis tasks. This review first briefly introduces several popular deep learning architectures, and then summarizes and thoroughly discusses their applications in various specific tasks in US image analysis, such as classification, detection, and segmentation. Finally, the open challenges and potential trends of the future application of deep learning in medical US image analysis are discussed.

Keywords

Deep learning / Medical ultrasound analysis / Classification / Segmentation / Detection

References

[[1]] Reddy U.M., Filly R.A., Copel J.A.. Prenatal imaging: ultrasonography and magnetic resonance imaging. Obstet Gynecol. 2008; 112(1): 145-157.

[[2]] Noble J.A., Boukerroui D.. Ultrasound image segmentation: a survey. IEEE Trans Med Imaging. 2006; 25(8): 987-1010.

[[3]] Salomon L.J., Winer N., Bernard J.P., Ville Y.. A score-based method for quality control of fetal images at routine second-trimester ultrasound examination. Prenat Diagn. 2008; 28(9): 822-827.

[[4]] Krizhevsky A., Sutskever I., Hinton G.E.. ImageNet classification with deep convolutional neural networks. Commun ACM. 2017; 60(6): 84-90.

[[5]] Wang G.. A perspective on deep imaging. IEEE Access. 2016; 4: 8914-8924.

[[6]] Russakovsky O., Deng J., Su H., Krause J., Satheesh S., Ma S., . Imagenet large scale visual recognition challenge. Int J Comput Vis. 2015; 115(3): 211-252.

[[7]] Salakhutdinov R.. Learning deep generative models. Annu Rev Stat Appl. 2015; 2(1): 361-385.

[[8]] LeCun Y., Bengio Y., Hinton G.. Deep learning. Nature. 2015; 521(7553): 436-444.

[[9]] Deng L., Yu D.. Deep learning: methods and applications. Found Trends. Signal Process. 2014; 7(3–4): 197-387.

[[10]] Deng L., Li J., Huang J.T., Yao K., Yu D., Seide F., . Recent advances in deep learning for speech research at Microsoft. In: Proceedings of 2013 IEEE International Conference on Acoustics, Speech and Signal Processing; 2013 May 26–31; Vancouver, BC, Canada. New York: IEEE; 2013. p. 8604-8608.

[[11]] Shen D., Wu G., Suk H.I.. Deep learning in medical image analysis. Annu Rev Biomed Eng. 2017; 19(1): 221-248.

[[12]] Greenspan H., Van Ginneken B., Summers R.M.. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imaging. 2016; 35(5): 1153-1159.

[[13]] Litjens G., Kooi T., Bejnordi B.E., Setio A.A.A., Ciompi F., Ghafoorian M., . A survey on deep learning in medical image analysis. Med Image Anal. 2017; 42: 60-88.

[[14]] Suzuki K.. Overview of deep learning in medical imaging. Radiological Phys Technol. 2017; 10(3): 257-273.

[[15]] Ker J., Wang L., Rao J., Lim T.. Deep learning applications in medical image analysis. IEEE Access. 2018; 6: 9375-9389.

[[16]] Anas E.M.A., Seitel A., Rasoulian A., John P.S., Pichora D., Darras K., . Bone enhancement in ultrasound using local spectrum variations for guiding percutaneous scaphoid fracture fixation procedures. Int J CARS. 2015; 10(6): 959-969.

[[17]] Hiramatsu Y., Muramatsu C., Kobayashi H., Hara T., Fujita H.. Automated detection of masses on whole breast volume ultrasound scanner: false positive reduction using deep convolutional neural network. In: Proceedings of the SPIE Medical Imaging; 2017 Feb 11–16; Orlando, FL, USA. Bellingham: SPIE; 2017.

[[18]] Bian C., Lee R., Chou Y., Cheng J.. Boundary regularized convolutional neural network for layer parsing of breast anatomy in automated whole breast ultrasound. In: editor. Medical image computing and computer-assisted intervention—MICCAI 2017. Berlin: Springer; 2017. p. 259-266.

[[19]] Shi J., Zhou S., Liu X., Zhang Q., Lu M., Wang T.. Stacked deep polynomial network based representation learning for tumor classification with small ultrasound image dataset. Neurocomputing. 2016; 194: 87-94.

[[20]] Azizi S., Imani F., Zhuang B., Tahmasebi A., Kwak J.T., Xu S., . Ultrasound-based detection of prostate cancer using automatic feature selection with deep belief networks. In: editor. Medical image computing and computer-assisted intervention—MICCAI 2015. Berlin: Springer; 2015. p. 70-77.

[[21]] Yang X., Yu L., Wu L., Wang Y., Ni D., Qin J., . Fine-grained recurrent neural networks for automatic prostate segmentation in ultrasound images. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence; 2017 Feb 4–9; San Francisco, CA. USA: AAAI Press; 2017. p. 1633-1639.

[[22]] Wu K., Chen X., Ding M.. Deep learning based classification of focal liver lesions with contrast-enhanced ultrasound. Optik. 2014; 125(15): 4057-4063.

[[23]] Ghesu F.C., Georgescu B., Zheng Y., Hornegger J., Comaniciu D.. Marginal space deep learning: efficient architecture for detection in volumetric image data. In: editor. Medical image computing and computer-assisted intervention. Berlin: Springer; 2015. p. 710-718.

[[24]] Pereira F., Bueno A., Rodriguez A., Perrin D., Marx G., Cardinale M., . Automated detection of coarctation of aorta in neonates from two-dimensional echocardiograms. J Med Imaging. 2017; 4(1): 014502.

[[25]] Sombune P., Phienphanich P., Phuechpanpaisal S., Muengtaweepongsa S., Ruamthanthong A., Tantibundhit C.. Automated embolic signal detection using deep convolutional neural network. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Piscataway: IEEE; 2017. p. 3365-3368.

[[26]] Milletari F., Ahmadi S.A., Kroll C., Hennersperger C., Tombari C., Shah A., . Robust segmentation of various anatomies in 3D ultrasound using hough forests and learned data representations. In: editor. Medical image computing and computer-assisted intervention. Berlin: Springer; 2015. p. 111-118.

[[27]] Lekadir K., Galimzianova A., Betriu A., Del Mar Vila M, Igual L., Rubin D.L., . A convolutional neural network for automatic characterization of plaque composition in carotid ultrasound. IEEE J Biomed Health Inform. 2017; 21(1): 48-55.

[[28]] Shin J., Tajbakhsh N., Hurst R.T., Kendall C.B., Liang J.. Automating carotid intima-media thickness video interpretation with convolutional neural networks. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 27–30; Las Vegas, NV, USA. Piscataway: IEEE; 2016. p. 2526-2535.

[[29]] Ma J., Wu F., Jiang T., Zhu J., Kong D.. Cascade convolutional neural networks for automatic detection of thyroid nodules in ultrasound images. Med Phys. 2017; 44(5): 1678-1691.

[[30]] Smistad E., Løvstakken L.. Vessel detection in ultrasound images using deep convolutional neural networks. In: , editor. Deep learning and data labeling for medical applications. Berlin: Springer; 2016. p. 30-38.

[[31]] Su S., Gao Z., Zhang H., Lin Q., Hao W.K., Li S.. Detection of lumen and media-adventitia borders in IVUS images using sparse auto-encoder neural network. In: Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21, Melbourne, Australia. Piscataway: IEEE; 2017. p. 1120-1124.

[[32]] Yaqub M., Kelly B., Papageorghiou A.T., Noble J.A.. A deep learning solution for automatic fetal neurosonographic diagnostic plane verification using clinical standard constraints. Ultrasound Med Biol. 2017; 43(12): 2925-2933.

[[33]] Huang W., Bridge C.P., Noble J.A., Zisserman A.. Temporal HeartNet: towards human-level automatic analysis of fetal cardiac screening video. In: Proceedings of 2017 IEEE Medical Image Computing and Computer-Assisted Intervention; 2017 Sep 11–13; Quebec City, Canada. Berlin: Springer; 2017. p. 341-349.

[[34]] Gao Y., Noble J.A.. Detection and characterization of the fetal heartbeat in free-hand ultrasound sweeps with weakly-supervised two-streams convolutional networks. In: Proceedings of 2017 IEEE Medical Image Computing and Computer-Assisted Intervention; 2017 Sep 11–13; Quebec City, Canada. Berlin: Springer; 2017. p. 305-313.

[[35]] Qi H., Collins S., Noble A.. Weakly supervised learning of placental ultrasound images with residual networks. In: editor. Medical image understanding and analysis. Berlin: Springer; 2017. p. 98-108.

[[36]] Chen H., Zheng Y., Park J.H., Heng P.A., Zhou K.. Iterative multi-domain regularized deep learning for anatomical structure detection and segmentation from ultrasound images. In: Proceedings of 2016 IEEE Medical Image Computing and Computer-Assisted Intervention; 2016 Oct 17–21; Athens, Greece. Berlin: Springer; 2016. p. 487-495.

[[37]] Ravishankar H., Prabhu S.M., Vaidya V., Singhal N.. Hybrid approach for automatic segmentation of fetal abdomen from ultrasound images using deep learning. In: Proceedings of 2016 IEEE 13th International Symposium on Biomedical Imaging; 2016 Jun 13–16; Prague, Czech Republic. Piscataway: IEEE; 2016. p. 779-782.

[[38]] Zhang Y., Ying M.T.C., Yang L., Ahuja A.T., Chen D.Z., . Coarse-to-fine stacked fully convolutional nets for lymph node segmentation in ultrasound images. In: Proceedings of 2016 IEEE International Conference on Bioinformatics and Biomedicine; 2016 Dec 15–18; Shenzhen, China. Piscataway: IEEE; 2016. p. 443-448.

[[39]] Ravishankar H., Venkataramani R., Thiruvenkadam S., Sudhakar P., Vaidya V.. Learning and incorporating shape models for semantic segmentation. In: Proceedings of 2017 IEEE Medical Image Computing and Computer-Assisted Intervention; 2017 Sep 11–13; Quebec City, Canada. Piscataway: IEEE; 2017. p. 203-211.

[[40]] Hetherington J., Lessoway V., Gunka V., Abolmaesumi P., Rohling R.. SLIDE: automatic spine level identification system using a deep convolutional neural network. Int J CARS. 2017; 12(7): 1189-1198.

[[41]] Golan D., Donner Y., Mansi C., Jaremko J., Ramachandran M.. Fully automating Graf’s method for DDH diagnosis using deep convolutional neural networks. In: , editor. Deep learning and data labeling for medical applications. Proceedings of International Workshops on DLMIA and LABELS; 2016 Oct 21; Athens, Greece. Berlin: Springer; 2016. p. 130-141.

[[42]] Hareendranathan A.R., Zonoobi D., Mabee M., Cobzas D., Punithakumar K., Noga M.L., . Toward automatic diagnosis of hip dysplasia from 2D ultrasound. In: Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourne, Australia. 2017. p. 982-985.

[[43]] Burlina P., Billings S., Joshi N., Albayda J.. Automated diagnosis of myositis from muscle ultrasound: exploring the use of machine learning and deep learning methods. PLoS ONE. 2017; 12(8): e0184059.

[[44]] Hafiane A, Vieyres P, Delbos A. Deep learning with spatiotemporal consistency for nerve segmentation in ultrasound images. 2017. arXiv:1706.05870.

[[45]] Fasel I., Berry J.. Deep belief networks for real-time extraction of tongue contours from ultrasound during speech. In: Proceedings of 2010 20th International Conference on Pattern Recognition; 2010 Aug 23–26; Istanbul, Turkey. 2010. p. 1493-1496.

[[46]] Jaumard-Hakoun A, Xu K, Roussel-Ragot P, Dreyfus G, Denby B. Tongue contour extraction from ultrasound images based on deep neural network. 2016. arXiv:1605.05912.

[[47]] Xu K., Roussel P., Csapó T.G., Denby B.. Convolutional neural network-based automatic classification of midsagittal tongue gestural targets using B-mode ultrasound images. J Acoust Soc Am. 2017; 141(6): EL531-EL537.

[[48]] Chi J., Walia E., Babyn P., Wang J., Groot G., Eramian M.. Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network. J Digit Imaging. 2017; 30(4): 477-486.

[[49]] Cheng P.M., Malhi H.S.. Transfer learning with convolutional neural networks for classification of abdominal ultrasound images. J Digit Imaging. 2017; 30(2): 234-243.

[[50]] Li Y., Xu R., Ohya J., Iwata H.. Automatic fetal body and amniotic fluid segmentation from fetal ultrasound images by encoder-decoder network with inner layers. In: Proceedings of 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2017 Jul 11–15; Seogwipo, Korea. Piscataway: IEEE; 2017. p. 1485-1488.

[[51]] Fenster A., Downey D.B., Cardinal H.N.. Three-dimensional ultrasound imaging. Phys Med Biol. 2001; 46(5): R67-R99.

[[52]] Ghesu F.C., Krubasik E., Georgescu B., Singh V., Zheng Y., Hornegger J., . Marginal space deep learning: efficient architecture for volumetric image parsing. IEEE Trans Med Imaging. 2016; 35(5): 1217-1228.

[[53]] Akkus Z., Galimzianova A., Hoogi A., Rubin D.L., Erickson B.J.. Deep learning for brain MRI segmentation: state of the art and future directions. J Digit Imaging. 2017; 30(4): 449-459.

[[54]] Xing F., Xie Y., Su H., Liu F., Yang L.. Deep learning in microscopy image analysis: a survey. IEEE Trans Neural Networks Learn Syst. 2017; 29(10): 4550-4568.

[[55]] Xian M., Zhang Y., Cheng H.D., Xu F., Zhang B., Ding J.. Automatic breast ultrasound image segmentation: a survey. Pattern Recognit. 2018; 79: 340-355.

[[56]] He K., Sun J.. Convolutional neural networks at constrained time cost. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition; 2015 Oct 7–12; Boston, MA, USA. Piscataway: IEEE; 2015. p. 5353-5360.

[[57]] Milletari F., Ahmadi S.A., Kroll C., Plate A., Rozanski V., Maiostre J., . Hough-CNN: deep learning for segmentation of deep brain regions in MRI and ultrasound. Comput Vis Image Underst. 2017; 164: 92-102.

[[58]] Liu X., Song J.L., Wang S.H., Zhao J.W., Chen Y.Q.. Learning to diagnose cirrhosis with liver capsule guided ultrasound image classification. Sensors. 2017; 17(1): 149.

[[59]] Canziani A, Paszke A, Culurciello E. An analysis of deep neural network models for practical applications. 2016. arXiv:1605.07678.

[[60]] Chen H., Dou Q., Ni D., Cheng J., Qin J., Li S., . Automatic fetal ultrasound standard plane detection using knowledge transferred recurrent neural networks. In: Proceedings of 2015 IEEE Medical Image Computing and Computer-Assisted Intervention; 2015 Oct 5–9; Munich, Germany. Berlin: Springer; 2015. p. 507-514.

[[61]] Bengio Y., Simard P., Frasconi P.. Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw. 1994; 5(2): 157-166.

[[62]] Hochreiter S., Schmidhuber J.. Long short-term memory. Neural Comput. 1997; 9(8): 1735-1780.

[[63]] Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H. Learning phrase representations using RNN encoder-decoder for statistical machine translation. 2014. arXiv:1406.1078.

[[64]] Bengio Y., Courville A., Vincent P.. Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell. 2013; 35(8): 1798-1828.

[[65]] Vincent P., Larochelle H., Lajoie I., Bengio Y., Manzagol P.A.. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res. 2010; 11(12): 3371-3408.

[[66]] Hinton G.E.. A practical guide to training restricted boltzmann machines. In: editor. Neural networks: tricks of the trade. 2nd ed. Berlin: Springer; 2012. p. 599-619.

[[67]] Hinton G.E., Salakhutdinov R.R.. Reducing the dimensionality of data with neural networks. Science. 2006; 313(5786): 504-507.

[[68]] Tajbakhsh N., Shin J.Y., Gurudu S.R., Hurst R.T., Kendall C.B., Gotway M.B., . Convolutional neural networks for medical image analysis: full training or fine tuning?. IEEE Trans Med Imaging. 2016; 35(5): 1299-1312.

[[69]] Duchi J., Hazan E., Singer Y.. Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res. 2011; 12(7): 2121-2159.

[[70]] Sutskever I., Martens J., Dahl G., Hinton G.. On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th International Conference on International Conference on Machine Learning; 2013 Jun 16–21; Atlanta, GA, USA. JMLR; 2013. p. 1139-1147.

[[71]] Nair V., Hinton G.E.. Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning; 2010 Jun 21–24; Haifa, Israel. Piscataway: Omnipress; 2010. p. 807-814.

[[72]] Glorot X., Bordes A., Bengio Y.. Deep sparse rectifier neural networks. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics; 2011 Apr 11–13; Ft. Lauderdale, FL, USA. JMLR; 2011. p. 315-323.

[[73]] Goodfellow I.J., Warde-Farley D., Mirza M., Courville A., Bengio Y.. Maxout networks. In: Proceedings of the 30th International Conference on Machine Learning; 2013 Jun 16–21; Atlanta, GA, USA. JMLR; 2013. p. 1319-1327.

[[74]] Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR. Improving neural networks by preventing co-adaptation of feature detectors. 2012. arXiv: 1207.0580.

[[75]] Ioffe S., Szegedy C.. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning; 2015 Jul 6–11; Lille, France. JMLR; 2015. p. 448-456.

[[76]] Pan S.J., Yang Q.. A survey on transfer learning. IEEE Trans Knowl Data Eng. 2010; 22(10): 1345-1359.

[[77]] Azizi S., Mousavi P., Yan P., Tahmasebi A., Kwak J.T., Xu S., . Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection. Int J CARS. 2017; 12(7): 1111-1121.

[[78]] Chen H., Ni D., Qin J., Li S., Yang X., Wang T., . Standard plane localization in fetal ultrasound via domain transferred deep neural networks. IEEE J Biomed Health Inform. 2015; 19(5): 1627-1636.

[[79]] Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., . Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia; 2014 Nov 3–7; New York, NY, USA. New York: ACM; 2014. p. 675-678.

[[80]] Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, et al. Tensorflow: large-scale machine learning on heterogeneous distributed systems. 2016. arXiv: 1603.04467.

[[81]] Bastien F, Lamblin P, Pascanu R, Bergstra J, Goodfellow L, Bergeron A, et al. Theano: new features and speed improvements. 2012. arXiv: 1211.5590.

[[82]] Collobert R., Kavukcuoglu K., Farabet C.. Torch7: a matlab-like environment for machine learning.

[[83]] Chen T, Li M, Li Y, Lin M, Wang N, Wang M. MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems. 2015. arXiv: 1512.01274.

[[84]] Bahrampour S, Ramakrishnan N, Schott L, Shah M. Comparative study of deep learning software frameworks. 2015. arXiv: 1511.06435.

[[85]] Giger M.L., Chan H.P., Boone J.. Anniversary paper: history and status of CAD and quantitative image analysis: the role of Medical Physics and AAPM. Med Phys. 2008; 35(12): 5799-5820.

[[86]] Jamieson A., Drukker K., Giger M.. Breast image feature learning with adaptive deconvolutional networks.

[[87]] Liu X., Shi J., Zhang Q.. Tumor classification by deep polynomial network and multiple kernel learning on small ultrasound image dataset. In: Proceedings of the 6th International Workshop on Machine Learning in Medical Imaging; 2015 Oct 5; Munich, Germany. Berlin: Springer; 2015. p. 313-320.

[[88]] Cheng J.Z., Ni D., Chou Y.H., Qin J., Tiu C.M., Chang Y.C., . Computer-aided diagnosis with deep learning architecture: applications to breast lesions in US images and pulmonary nodules in CT scans. Sci Rep. 2016; 6(1): 24454.

[[89]] Zhang Q., Xiao Y., Dai W., Suo J., Wang C., Shi J., . Deep learning based classification of breast tumors with shear-wave elastography. Ultrasonics. 2016; 72: 150-157.

[[90]] Han S., Kang H.K., Jeong J.Y., Park M.H., Kim W., Bang W.C., . A deep learning framework for supporting the classification of breast lesions in ultrasound images. Phys Med Biol. 2017; 62(19): 7714-7728.

[[91]] Antropova N., Huynh B.Q., Giger M.L.. A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets. Med Phys. 2017; 44(10): 5162-5171.

[[92]] Ferlay J., Shin H.R., Bray F., Forman D., Mathers C., Parkin D.M.. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. Int J Cancer. 2010; 127(12): 2893-2917.

[[93]] Guo L., Wang D., Xu H., Qian Y., Wang C., Zheng X., . CEUS-based classification of liver tumors with deep canonical correlation analysis and multi-kernel learning. In: Proceedings of 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2017 Jul 11–15; Seogwipe, Korea. Piscataway: IEEE; 2017. p. 1748-1751.

[[94]] Meng D., Zhang L., Cao G., Cao W., Zhang G., Hu B.. Liver fibrosis classification based on transfer learning and FCNet for ultrasound images. IEEE Access. 2017; 5: 5804-5810.

[[95]] Ma J., Wu F., Zhu J., Xu D., Kong D.. A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics. 2017; 73: 221-230.

[[96]] Liu T., Xie S., Yu J., Niu L., Sun W.D.. Classification of thyroid nodules in ultrasound images using deep model based transfer learning and hybrid features. In: Proceedings of 2017 IEEE International Conference on Acoustics, Speech and Signal Processing; 2017 Jun 19; New Orleans, LA, USA. Piscataway: IEEE; 2017. p. 919-923.

[[97]] Liu T., Xie S., Zhang Y., Yu J., Niu L., Sun W.. Feature selection and thyroid nodule classification using transfer learning. In: Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourne, Australia. Piscataway: IEEE; 2017. p. 1096-1099.

[[98]] Dudley N.J., Chapman E.. The importance of quality management in fetal measurement. Ultrasound Obstet Gynecol. 2002; 19(2): 190-196.

[[99]] Wu L., Cheng J.Z., Li S., Lei B., Wang T., Ni D.. FUIQA: fetal ultrasound image quality assessment with deep convolutional networks. IEEE Trans Cybern. 2017; 47(5): 1336-1349.

[[100]] Jang J, Kwon JY, Kim B, Lee SM, Park Y, Seo JK. CNN-based estimation of abdominal circumference from ultrasound images. 2017. arXiv: 1702.02741.

[[101]] Gao Y., Maraci M.A., Noble J.A.. Describing ultrasound video content using deep convolutional neural networks. In: Proceedings of 2016 IEEE 13th International Symposium on Biomedical Imaging. Piscataway: IEEE; 2016. p. 787-790.

[[102]] Sundaresan V., Bridge C.P., Ioannou C., Noble A.. Automated characterization of the fetal heart in ultrasound images using fully convolutional neural networks. In: 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourne, Australia. Piscataway: IEEE; 2017. p. 671-674.

[[103]] Perrin D.P., Bueno A., Rodriguez A., Marx G.R., Del Nido P.J.. Application of convolutional artificial neural networks to echocardiograms for differentiating congenital heart diseases in a pediatric population.

[[104]] Yu Z., Ni D., Chen S., Li S., Wang T., Lei B.. Fetal facial standard plane recognition via very deep convolutional networks. In: Proceedings of 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2016 Aug 16–20; Orlando, FL, USA. Piscataway: IEEE; 2016. p. 627-630.

[[105]] Yu Z., Tan E.L., Ni D., Qin J., Chen S., Li S., . A deep convolutional neural network-based framework for automatic fetal facial standard plane recognition. IEEE J Biomed Health Inform. 2018; 22(3): 874-885.

[[106]] Azizi S., Imani F., Ghavidel S., Tahmasebi A., Kwak J.T., Xu S., . Detection of prostate cancer using temporal sequences of ultrasound data: a large clinical feasibility study. Int J CARS. 2016; 11(6): 947-956.

[[107]] Azizi S., Bayat S., Yan P., Tahmasebi A., Nir G., Kwak J.T., . Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks and tissue mimicking simulations. Int J CARS. 2017; 12(8): 1293-1305.

[[108]] Yap M.H., Pons G., Martí J., Ganau S., Sentis M., Zwiggelaar R., . Automated breast ultrasound lesions detection using convolutional neural networks. IEEE J Biomed Health Inform. 2018; 22(4): 1218-1226.

[[109]] Cao Z., Duan L., Yang G., Yue T., Chen Q., Fu C., . Breast tumor detection in ultrasound images using deep learning. In: editor. Patch-based techniques in medical imaging. Berlin: Springer; 2017. p. 121-128.

[[110]] Girshick R.. Fast R-CNN. In: Proceedings of 2015 IEEE International Conference on Computer Vision; 2015 Dec 7–13; Santiago, Chile. Piscataway: IEEE; 2015. p. 1440-1448.

[[111]] Ren S., He K., Girshick R., Sun J.. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell. 2017; 39(6): 1137-1149.

[[112]] Redmon J., Divvala S., Girshick R., Farhadi A.. You only look once: unified, real-time object detection. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 27–30; Las Vegas, NV, USA. Piscataway: IEEE; 2016. p. 779-788.

[[113]] Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C., . SSD: single shot multibox detector. In: Proceedings of the European Conference on Computer Vision; 2016 Oct 11–14; Amsterdam, The Netherlands. Berlin: Springer; 2016. p. 21-37.

[[114]] Lipsanen A., Parkkinen S., Khabbal J., Mäkinen P., Peräniemi S., Hiltunen M., . KB-R7943, an inhibitor of the reverse Na⁺/Ca²⁺ exchanger, does not modify secondary pathology in the thalamus following focal cerebral stroke in rats. Neurosci Lett. 2014; 580: 173-177.

[[115]] Yang X., Ni D., Qin J., Li S., Wang T., Chen S., . Standard plane localization in ultrasound by radial component. In: Proceedings of 2014 IEEE 11th International Symposium on Biomedical Imaging; 2014 Apr 29–May 2; Beijing, China. Piscataway: IEEE; 2014. p. 1180-1183.

[[116]] Ni D., Li T., Yang X., Qin J., Li S., Chin C., . Selective search and sequential detection for standard plane localization in ultrasound. In: editor. Abdominal imaging, computation and clinical applications. Berlin: Springer; 2013. p. 203-211.

[[117]] Baumgartner C.F., Kamnitsas K., Matthew J., Smith S., Kainz B., Rueckert D., . Real-time standard scan plane detection and localisation in fetal ultrasound using fully convolutional neural networks. In: Proceedings of 2016 IEEE Medical Image Computing and Computer-Assisted Intervention; 2016 Oct 17–21; Athens, Greece. Berlin: Spring; 2016. p. 203-211.

[[118]] Baumgartner C.F., Kamnitsas K., Matthew J., Fletcher T.P., Smith S., Koch L.M., . SonoNet: real-time detection and localisation of fetal standard scan planes in freehand ultrasound. IEEE Trans Med Imaging. 2017; 36(11): 2204-2215.

[[119]] Chen H., Ni D., Yang X., Li S., Heng P.A.. Fetal abdominal standard plane localization through representation learning with knowledge transfer. In: editor. Machine learning in medical imaging. Berlin: Springer; 2014. p. 125-132.

[[120]] Chen H., Wu L., Dou Q., Qin J., Li S., Cheng J.Z., . Ultrasound standard plane detection using a composite neural network framework. IEEE Trans Cybern. 2017; 47(6): 1576-1586.

[[121]] Dezaki F.T., Dhungel N., Abdi A., Luong C., Tsang T., Jue J., . Deep residual recurrent neural networks for characterization of cardiac cycle phase from echocardiograms. In: editor. Deep learning in medical image analysis and multimodal learning for clinical decision support. Berlin: Springer; 2017. p. 100-108.

[[122]] Sofka M., Milletari F., Jia J., Rothberg A.. Fully convolutional regression network for accurate detection of measurement points. In: editor. Deep learning in medical image analysis and multimodal learning for clinical decision support. Berlin: Springer; 2017. p. 258-266.

[[123]] Ghesu F.C., Georgescu B., Mansi T., Neumann D., Hornegger J., Comaniciu D.. An artificial agent for anatomical landmark detection in medical images. In: Proceedings of 2016 IEEE Medical Image Computing and Computer-Assisted Intervention; 2016 Oct 17–21; Athens, Greece. Berlin: Springer; 2016. p. 229-237.

[[124]] Nascimento J.C., Carneiro G.. Multi-atlas segmentation using manifold learning with deep belief networks. In: Proceedings of 2016 IEEE 13th International Symposium on Biomedical Imaging; 2016 Apr 13–16; Prague, Czech Republic. Piscataway: IEEE; 2016. p. 867-871.

[[125]] Ma J., Wu F., Jiang T., Zhao Q., Kong D.. Ultrasound image-based thyroid nodule automatic segmentation using convolutional neural networks. Int J CARS. 2017; 12(11): 1895-1910.

[[126]] Singhal N., Mukherjee S., Perrey C.. Automated assessment of endometrium from transvaginal ultrasound using Deep Learned Snake. In: Proceedings of the 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourn, Australia. Piscataway: IEEE; 2017. p. 83-86.

[[127]] Bernard O., Touil B., Gelas A., Prost R., Friboulet D.. A RBF-Based multiphase level set method for segmentation in echocardiography using the statistics of the radiofrequency signal. In: Proceedings of 2007 IEEE International Conference on Image Processing; 2007 Oct 16–19; San Antonio, TX, USA. Piscataway: IEEE; 2007.

[[128]] Jacob G., Noble J.A., Behrenbruch C., Kelion A.D., Banning A.P.. A shape-space-based approach to tracking myocardial borders and quantifying regional left-ventricular function applied in echocardiography. IEEE Trans Med Imaging. 2002; 21(3): 226-238.

[[129]] Carneiro G., Nascimento J., Freitas A.. Robust left ventricle segmentation from ultrasound data using deep neural networks and efficient search methods. In: Proceedings of 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro; 2010 Apr 14–17; Rotterdam, The Netherlands. Piscataway: IEEE; 2010. p. 1085-1088.

[[130]] Carneiro G., Nascimento J.C.. Multiple dynamic models for tracking the left ventricle of the heart from ultrasound data using particle filters and deep learning architectures. In: Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition; 2010 Jun 13–18. San Francisco, CA, USA. Piscataway: IEEE; 2010. p. 2815-2822.

[[131]] Carneiro G., Nascimento J.C.. Incremental on-line semi-supervised learning for segmenting the left ventricle of the heart from ultrasound data. In: Proceedings of 2011 International Conference on Computer Vision; 2011 Nov 6–13; Barcelona, Spain. Piscataway: IEEE; 2011. p. 1700-1707.

[[132]] Carneiro G., Nascimento J.C.. The use of on-line co-training to reduce the training set size in pattern recognition methods: application to left ventricle segmentation in ultrasound. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition; 2012 Jun 16–21; Providence, RI, USA. Piscataway: IEEE; 2012. p. 948-955.

[[133]] Carneiro G., Nascimento J.C., Freitas A.. The segmentation of the left ventricle of the heart from ultrasound data using deep learning architectures and derivative-based search methods. IEEE Trans Image Process. 2012; 21(3): 968-982.

[[134]] Carneiro G., Nascimento J.C.. Combining multiple dynamic models and deep learning architectures for tracking the left ventricle endocardium in ultrasound data. IEEE Trans Pattern Anal Mach Intell. 2013; 35(11): 2592-2607.

[[135]] Nascimento J.C., Carneiro G.. Deep learning on sparse manifolds for faster object segmentation. IEEE Trans Image Process. 2017; 26(10): 4978-4990.

[[136]] Nascimento J.C., Carneiro G.. Non-rigid segmentation using sparse low dimensional manifolds and deep belief networks. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition; 2014 Jun 23–28; Columbus, OH, USA. Piscataway: IEEE; 2014. p. 288-295.

[[137]] Raynaud C., Langet H., Amzulescu M.S., Saloux E., Bertrand H., Allain P., . Handcrafted features vs. ConvNets in 2D echocardiographic images. In: Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourne, Australia. Piscataway: IEEE; 2017. p. 1116-1119.

[[138]] Xiong X., Torre F.D.L.. Global supervised descent method. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition; 2015 Jun 7–12; Boston, MA, USA. Piscataway: IEEE; 2015. p. 2664-2673.

[[139]] Yu L., Guo Y., Wang Y., Yu J., Chen P.. Segmentation of fetal left ventricle in echocardiographic sequences based on dynamic convolutional neural networks. IEEE Trans Biomed Eng. 2017; 64(8): 1886-1895.

[[140]] Baka N., Leenstra S., van Walsum T.. Ultrasound aided vertebral level localization for lumbar surgery. IEEE Trans Med Imaging. 2017; 36(10): 2138-2147.

[[141]] Wu L., Xin Y., Li S., Wang T., Heng P., Ni D.. Cascaded fully convolutional networks for automatic prenatal ultrasound image segmentation. In: Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourne, Australia. Piscataway: IEEE; 2017. p. 663-666.

[[142]] Tu Z., Bai X.. Auto-context and its application to high-level vision tasks and 3D brain image segmentation. IEEE Trans Pattern Anal Mach Intell. 2010; 32(10): 1744-1757.

[[143]] Anas E.M.A., Nouranian S., Mahdavi S.S., Spadinger I., Morris W.J., Salcudean S.E., . Clinical target-volume delineation in prostate brachytherapy using residual neural networks. In: Proceedings of 2017 IEEE Medical Image Computing and Computer-Assisted Intervention; 2017 Sep 11–13; Quebec City, Canada. Piscataway: IEEE; 2017. p. 365-373.

[[144]] Zheng Y., Liu D., Georgescu B., Nguyen H., Comaniciu D.. 3D deep learning for efficient and robust landmark detection in volumetric data. In: Proceedings of 2015 IEEE Medical Image Computing and Computer-Assisted Intervention; 2015 Oct 5–9; Munich, Germany. Piscataway: IEEE; 2015. p. 565-572.

[[145]] Pesteie M., Lessoway V., Abolmaesumi P., Rohling R.N.. Automatic localization of the needle target for ultrasound-guided epidural injections. IEEE Trans Med Imaging. 2018; 37(1): 81-92.

[[146]] Nie S., Yu J., Chen P., Wang Y., Zhang J.Q.. Automatic detection of standard sagittal plane in the first trimester of pregnancy using 3-D ultrasound data. Ultrasound Med Biol. 2017; 43(1): 286-300.

[[147]] Nie S., Yu J., Chen P., Zhang J., Wang Y.. A novel method with a deep network and directional edges for automatic detection of a fetal head. In: Proceedings of 2015 the 23rd European Signal Processing Conference; 2015 Aug 31–Sep 4; Nice, France. Piscataway: IEEE; 2015. p. 654-658.

[[148]] Pourtaherian A., Zanjani F.G., Zinger S., Mihajlovic N., Ng G., Korsten H., . Improving needle detection in 3D ultrasound using orthogonal-plane convolutional networks. In: Proceedings of 2017 IEEE Medical Image Computing and Computer-Assisted Intervention; 2017 Sep 11–13; Quebec City, Canada. Piscataway: IEEE; 2017. p. 610-618.

[[149]] Zheng Y., Barbu A., Georgescu B., Scheuering M., Comaniciu D.. Four-chamber heart modeling and automatic segmentation for 3-D cardiac CT volumes using marginal space learning and steerable features. IEEE Trans Med Imaging. 2008; 27(11): 1668-1681.

[[150]] Looney P., Stevenson G.N., Nicolaides K.H., Plasencia W., Molloholli M., Natsis S., . Automatic 3D ultrasound segmentation of the first trimester placenta using deep learning. In: Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourne, Australia. Piscataway: IEEE; 2017. p. 279-282.

[[151]] Kamnitsas K., Ledig C., Newcombe V.F.J., Simpson J.P., Kane A.D., Menon D.K., . Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal. 2017; 36: 61-78.

[[152]] Yang X., Yu L., Li S., Wang X., Wang N., Qin J., . Towards automatic semantic segmentation in volumetric ultrasound. In: Proceedings of 2017 IEEE Medical Image Computing and Computer-Assisted Intervention; 2017 Sep 11–13; Quebec City, Canada. Piscataway: IEEE; 2017. p. 711-719.

[[153]] Schmidt-Richberg A., Brosch T., Schadewaldt N., Klinder T., Caballaro A., Salim I., . Abdomen segmentation in 3D fetal ultrasound using CNN-powered deformable models. In: Proceedings of the 4th International Workshop on Fetal and Infant Image Analysis; 2017 Sep 14; Quebec City, Canada. Piscataway: IEEE; 2017. p. 52-61.

[[154]] Amit G., Ben-Ari R., Hadad O., Monovich E., Granot N., Hashoul S.. Classification of breast MRI lesions using small-size training sets: comparison of deep learning approaches. In: Proceedings of SPIE Medical Imaging: Computer-Aided Diagnosis; 2017 Mar 3; Orlando, Florida. Bellingham: SPIE; 2017.

[[155]] Shin H.C., Roth H.R., Gao M., Lu L., Xu Z., Nogues I., . Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging. 2016; 35(5): 1285-1298.

[[156]] Hadad O., Bakalo R., Ben-Ari R., Hashoul S., Amit G.. Classification of breast lesions using cross-modal deep learning. In: Proceedings of 2017 IEEE 14th International Symposium on Biomedical Imaging; 2017 Apr 18–21; Melbourne, Australia. Piscataway: IEEE; 2017. p. 109-112.

[[157]] Lisa T., Jude S.. Transfer learning. In: editor. Handbook of research on machine learning applications and trends: algorithms, methods, and techniques. Hershey: IGI Global; 2010. p. 242-264.

This work was supported in part by the National Natural Science Foundation of China (61571304, 81571758, and 61701312), in part by the National Key Research and Development Program of China (2016YFC0104703), in part by the Medical Scientific Research Foundation of Guangdong Province, China (B2018031), and in part by the Shenzhen Peacock Plan (KQTD2016053112051497).

Shengfeng Liu, Yi Wang, Xin Yang, Baiying Lei, Li Liu, Shawn Xiang Li, Dong Ni, and Tianfu Wang declare that they have no conflict of interest or financial conflicts to disclose.

RIGHTS & PERMISSIONS

2019 Chinese Academy of Engineering