Rail track condition monitoring: a review on deep learning approaches

Abstract

Rail track is a critical component of rail systems. Accidents or interruptions caused by rail track anomalies usually possess severe outcomes. Therefore, rail track condition monitoring is an important task. Over the past decade, deep learning techniques have been rapidly developed and deployed. In the paper, we review the existing literature on applying deep learning to rail track condition monitoring. Potential challenges and opportunities are discussed for the research community to decide on possible directions. Two application cases are presented to illustrate the implementation of deep learning to rail track condition monitoring in practice before we conclude the paper.

Keywords

Rail track maintenance / condition monitoring / anomaly detection and classification / deep learning

References

[1] Cannon DF,Grassie SL.Rail defects: an overview.Fatigue Fract Eng M2003;26:865-86

[2] Track circuit monitoring tool: standardization and deployment at CTA. Available from: http://www.trb.org/Main/Blurbs/177054.aspx [Last accessed on 5 Jan 2022]

[3] Rail Defects Handbook. Available from: https://extranet.artc.com.au/docs/eng/track-civil/guidelines/rail/RC2400.pdf [Last accessed on 5 Jan 2022]

[4] Dey A,Tenczynski L. Detection and evaluation of rail defects with non-destructive testing methods. Available from: https://www.ndt.net/article/wcndt2016/papers/we1g4.pdf [Last accessed on 5 Jan 2022]

[5] Min Y,Dang J,Cheng T.Real time detection system for rail surface defects based on machine vision.J Image Video Proc2018;

[6] Serin G,Ozbayoglu AM.Review of tool condition monitoring in machining and opportunities for deep learning.Int J Adv Manuf Technol2020;109:953-74

[7] Zhao R,Chen Z,Wang P.Deep learning and its applications to machine health monitoring.Mech Syst Signal Process2019;115:213-37

[8] Fu J,Guo P.Condition monitoring of wind turbine gearbox bearing based on deep learning model.IEEE Access2019;7:57078-87

[9] LeCun Y,Hinton G.Deep learning.Nature2015;521:436-44

[10] Mcculloch WS.A logical calculus of the ideas immanent in nervous activity.Bull Math Biol.1943;5:115-33

[11] Rosenblatt F.The perceptron: a probabilistic model for information storage and organization in the brain.Psychol Rev1958;65:386-408

[12] Newell A.A step toward the understanding of information processes.Science1969;165:780-2

[13] Rodan A,Alqatawna J.Optimizing feedforward neural networks using biogeography based optimization for E-mail spam identification.IJCNS2016;9:19-28

[14] Robert HN.Theory of the backpropagation neural network.Proc 1989 IEEE IJCNN1989;1:593-605

[15] Lecun Y,Denker JS.Backpropagation applied to handwritten zip code recognition.Neural Comput1989;1:541-51

[16] Hochreiter S.Untersuchungen zu dynamischen neuronalen Netzen.Diploma: Technische Universität München1991;

[17] Hochreiter S.Long short-term memory.Neural Comput1997;9:1735-80

[18] Quinlan JR.Induction of decision trees.Mach Learn1986;1:81-106

[19] Cortes C.Support-vector networks.Mach Learn1995;20:273-97

[20] Freund Y.A decision-theoretic generalization of on-line learning and an application to boosting.J Comput Syst Sci1997;55:119-39

[21] Cristianini N.Support vector machines and kernel methods: the new generation of learning machines.Ai Magazine2002;23:31

[22] Breiman L.Random forests.Mach Learn2001;45:5-32

[23] Murphy K.An introduction to graphical models.Rap tech2001;96:1-19

[24] Hinton GE,Teh YW.A fast learning algorithm for deep belief nets.Neural Comput2006;18:1527-54

[25] Glorot X,Bengio Y. Deep sparse rectifier neural networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS); Fort Lauderdale, FL, USA. 2011. p. 315-23.

[26] Krizhevsky A,Hinton GE.ImageNet classification with deep convolutional neural networks.Commun ACM2017;60:84-90

[27] Koch G,Salakhutdinov R. Siamese neural networks for one-shot image recognition. Proceedings of the 32nd International Conference on Machine Learning; Lille, France. 2015.

[28] Ronneberger O,Brox T.U-Net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF, editors. Medical image computing and computer-assisted intervention - MICCAI 2015. Cham: Springer International Publishing; 2015. p. 234-41.

[29] Tan C,Kong T,Yang C. A survey on deep transfer learning. In: Kůrková V, Manolopoulos Y, Hammer B, Iliadis L, Maglogiannis I, editors. Artificial neural networks and machine learning - ICANN 2018. Cham: Springer; 2008. p. 270-9.

[30] Goodfellow I,Mirza M.Generative adversarial networks.Commun ACM2020;63:139-44

[31] Zeiler MD.Visualizing and understanding convolutional networks. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T, editors. Computer Vision - ECCV 2014. Cham: Springer International Publishing; 2014. p. 818-33.

[32] Szegedy C,Jia Y.Going deeper with convolutions. Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2015 Jun 7-12; Boston, MA. IEEE; 2005. p. 1-9.

[33] He K,Ren S.Deep residual learning for image recognition. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 Jun 27-30; Las Vegas, NV, USA. IEEE; 2016. p. 770-8.

[34] Targ S,Lyman K. Resnet in resnet: generalizing residual architectures. arXiv preprint arXiv:1603.08029.

[35] Zhang K,Han TX,Guo L.Residual networks of residual networks: multilevel residual networks.IEEE Trans Circuits Syst Video Technol2018;28:1303-14

[36] Zagoruyko S. Wide residual networks. arXiv preprint arXiv:1605.07146.

[37] Xie S,Dollár P,He K.Aggregated residual transformations for deep neural networks. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21-26; Honolulu, HI, USA. IEEE; 2017. p. 5987-95.

[38] Huang G,Van Der Maaten L.Densely connected convolutional networks. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21-26; Honolulu, HI, USA. IEEE; 2017. p. 2261-9.

[39] Huang G,Liu Z,Weinberger KQ.Deep networks with stochastic depth. In: Leibe B, Matas J, Sebe N, Welling M, editors. Computer vision - ECCV 2016. Cham: Springer International Publishing; 2016. p. 646-61.

[40] Simonyan K. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

[41] He K,Ren S.Spatial pyramid pooling in deep convolutional networks for visual recognition.IEEE Trans Pattern Anal Mach Intell2015;37:1904-16

[42] He K,Ren S.Delving deep into rectifiers: surpassing human-level performance on imagenet classification. Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV); 2015 Dec 7-13; Santiago, Chile. IEEE; 2015. p. 1026-34.

[43] Chollet F.Xception: deep learning with depthwise separable convolutions. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21-26; Honolulu, HI, USA. IEEE; 2017. p. 1800-7.

[44] Howard AG,Chen B. Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

[45] Larsson G,Shakhnarovich G. Fractalnet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648.

[46] Sermanet P,Zhang X,Fergus R. Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229.

[47] Girshick R,Darrell T.Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition; 2014 Jun 23-28; Columbus, OH, USA. IEEE; 2014. p. 580-7.

[48] Girshick R.Fast R-CNN. Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV); 2015 Dec 7-13; Santiago, Chile. IEEE; 2015. p. 1440-8.

[49] Ren S,Girshick R.Faster R-CNN: towards real-time object detection with region proposal networks.IEEE Trans Pattern Anal Mach Intell2017;39:1137-49

[50] Ouyang W,Wang X.DeepID-Net: deformable deep convolutional neural networks for object detection.IEEE Trans Pattern Anal Mach Intell2017;39:1320-34

[51] Dai J,He K. R-fcn: object detection via region-based fully convolutional networks. Available from: https://arxiv.org/pdf/1605.06409.pdf [Last accessed on 5 Jan 2022]

[52] Redmon J,Girshick R.You only look once: unified, real-time object detection. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016 Jun 27-30; Las Vegas, NV, USA. IEEE; 2016. p. 779-88.

[53] Liu W,Erhan D.SSD: Single Shot MultiBox Detector. In: Leibe B, Matas J, Sebe N, Welling M, editors. Computer vision - ECCV 2016. Cham: Springer International Publishing; 2016. p. 21-37.

[54] Shelhamer E,Darrell T.Fully convolutional networks for semantic segmentation.IEEE Trans Pattern Anal Mach Intell2017;39:640-51

[55] Noh H,Han B.Learning deconvolution network for semantic segmentation. Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV); 2015 Dec 7-13; Santiago, Chile. IEEE; 2015. p. 1520-8.

[56] Chen LC,Kokkinos I,Yuille AL.DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs.IEEE Trans Pattern Anal Mach Intell2018;40:834-48

[57] Liu W,Berg AC. Parsenet: looking wider to see better. arXiv preprint arXiv:1506.04579.

[58] Yu F. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122.

[59] Zhao H,Qi X,Jia J.Pyramid scene parsing network. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2017 Jul 21-26; Honolulu, HI, USA. IEEE; 2017. p. 6230-9.

[60] Fink O,Weidmann U.Predicting component reliability and level of degradation with complex-valued neural networks.2014;121:198-206

[61] Giben X,Chellappa R.Material classification and semantic segmentation of railway track images with deep convolutional neural networks. Proceedings of 2015 IEEE International Conference on Image Processing (ICIP); 2015 Sep 27-30; Quebec City, QC, Canada. IEEE; 2015. p. 621-5.

[62] Faghih-Roohi S,Núnez A,De Schutter B.Deep convolutional neural networks for detection of rail surface defects. Proceedings of 2016 International joint conference on neural networks (IJCNN); 2016 Jul 24-29; Vancouver, BC, Canada. IEEE; 2016. p. 2584-9.

[63] Bruin T, Verbert K, Babuska R. Railway track circuit fault diagnosis using recurrent neural networks.IEEE Trans Neural Netw Learn Syst2017;28:523-33

[64] Gibert X,Chellappa R.Deep multitask learning for railway track inspection.IEEE Trans Intell Transport Syst2017;18:153-64

[65] Zhang X,Wang Y,Hu H.An improved method of rail health monitoring based on CNN and multiple acoustic emission events. Proceedings of 2017 IEEE International Instrumentation and Measurement Technology Conference (I2MTC); 2017 May 22-25; Turin, Italy. IEEE; 2017. p. 1-6.

[66] Rao DJ,Ritika S. Siamese neural networks for one-shot detection of railway track switches. arXiv preprint arXiv:1712.08036.

[67] Mittal S.Vision based railway track monitoring using deep learning. arXiv preprint arXiv:1711.06423.

[68] Santur Y,Akin E.A new rail inspection method based on deep learning using laser cameras. Proceedings of 2017 International Artificial Intelligence and Data Processing Symposium (IDAP); 2017 Sep 16-17; Malatya, Turkey. IEEE; 2017. p. 1-6.

[69] Santur Y,Akin E.An adaptive fault diagnosis approach using pipeline implementation for railway inspection.2018;26:987-98

[70] Huang H,Zhang J,Kirsch C.Railway infrastructure defects recognition using fine-grained deep convolutional neural networks. Proceedings of 2018 Digital Image Computing: Techniques and Applications (DICTA); 2018 Dec 10-13; Canberra, ACT, Australia. IEEE; 2018. p. 1-8.

[71] Zhang X,Wang K.A new rail crack detection method using LSTM network for actual application based on AE technology.Appl Acoust2018;142:78-86

[72] Ye T,Song P.Automatic railway traffic object detection system using feature fusion refine neural network under shunting mode.Sensors (Basel)2018;18:1916

[73] Kang G,Yu L.Deep architecture for high-speed railway insulator surface defect detection: denoising autoencoder with multitask learning.IEEE Trans Instrum Meas2019;68:2679-90

[74] Liang Z,Liu L,Zheng K.Defect detection of rail surface with deep convolutional neural networks. Proceedings of 2018 13th World Congress on Intelligent Control and Automation (WCICA); 2018 Jul 4-8; Changsha, China. IEEE; 2018. p. 1317-22.

[75] Shang L,Wang J,Lei W.Detection of rail surface defects based on CNN image recognition and classification. Proceedings of 2018 20th International Conference on Advanced Communication Technology (ICACT); 2018 Feb 11-14; Chuncheon, Korea (South). IEEE; 2018. p. 45-51.

[76] Wang S,Du X,Ma Y.Rail fastener automatic recognition method in complex background. Proceedings of Tenth International Conference on Digital Image Processing (ICDIP 2018); 2018 Aug 9; Shanghai, China. International Society for Optics and Photonics; 2018. p. 1080625.

[77] Yanan S,Li L.Rail surface defect detection method based on yolov3 deep learning networks. Proceedings of 2018 Chinese Automation Congress (CAC); 2018 Nov 30-Dec 2; Xi’an, China. IEEE; 2018. p. 1563-8.2

[78] Xu X,Yang F.Railway subgrade defect automatic recognition method based on improved faster R-CNN.Sci Programming2018;2018:1-12

[79] Jamshidi A,Su Z.A decision support approach for condition-based maintenance of rails based on big data analysis.Transp Res Part C Emerg Technol2018;95:185-206

[80] Bukhsh Z, Saeed A, Stipanovic I, Doree AG. Predictive maintenance using tree-based classification techniques: a case of railway switches.Transp Res Part C Emerg Technol2019;101:35-54

[81] James A,Xulei Y.Tracknet - a deep learning based fault detection for railway track inspection. Proceedings of 2018 International Conference on Intelligent Rail Transportation (ICIRT); 2018 Dec 12-14; Singapore. IEEE; 2018. p. 1-5.

[82] Peng X.Rail suspension system fault detection using deep semi-supervised feature extraction with one-class data.PHM_CONF2018;

[83] Sun Y,Yang C.Railway joint detection using deep convolutional neural networks. Proceedings of 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE); 2019 Aug 22-26; Vancouver, BC, Canada. IEEE; 2019. p. 235-40.

[84] Yuan H,Liu S,Luo X.A deep convolutional neural network for detection of rail surface defect. Proceedings of 2019 IEEE Vehicle Power and Propulsion Conference (VPPC); 2019 Oct 14-17; Hanoi, Vietnam. IEEE; 2019. p. 1-4.

[85] Wang Y,Yu Z.An adaptive track segmentation algorithm for a railway intrusion detection system.Sensors (Basel)2019;19:2594 PMCID:PMC6603763

[86] Dong B,Wang J,Dai P.An end-to-end abnormal fastener detection method based on data synthesis. Proceedings of 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI); 2019 Nov 4-6; Portland, OR, USA. IEEE; 2019. p. 149-56.

[87] Ma S,Liu X.Deep learning for track quality evaluation of high-speed railway based on vehicle-body vibration prediction.IEEE Access2019;7:185099-107

[88] Jin X,Zhang H.DM-RIS: deep multimodel rail inspection system with improved MRF-GMM and CNN.IEEE Trans Instrum Meas2020;69:1051-65

[89] Li Z,Tang T.Fault diagnosis of railway point machines using the locally connected autoencoder.Appl Sci2019;9:5139

[90] Guo B,Zhu L.High-speed railway clearance intrusion detection with improved SSD network.Appl Sci2019;9:2981

[91] Liu J,Zou Q.Learning visual similarity for inspecting defective railway fasteners.IEEE Sensors J2019;19:6844-57

[92] Cui H,Hu Q.Real-time inspection system for ballast railway fasteners based on point cloud deep learning.IEEE Access2020;8:61604-14

[93] Haseeb M,Gräser A. A deep learning based autonomous distance estimation and tracking of multiple objects for improvement in safety and security in railways. Available from: https://www.bmvc2019.org/wp-content/ODRSS2019/ODRSS2019_P_5_Haseeb.pdf [Last accessed on 5 Jan 2022]

[94] Chen SX,Liu JC.Deep learning-based data anomaly detection in rail track inspection. Proceedings of 12th International Workshop on Structural Health Monitoring: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT); 2019; Stanford, USA. DEStech Publications Inc.; 2019. p. 3235-42.

[95] Jang J,Lim S,Kim J.Intelligent image-based railway inspection system using deep learning-based object detection and weber contrast-based image comparison.Sensors2019;19:4738 PMCID:PMC6864465

[96] Kuzmin EV,Plotnikov PO,Bashkin VA.Application of neural networks for recognizing rail structural elements in magnetic and eddy current defectograms.Aut Control Comp Sci2019;53:628-37

[97] Pahwa RS,Paul J.Faultnet: faulty rail-valves detection using deep learning and computer vision. Proceedings of 2019 IEEE Intelligent Transportation Systems Conference (ITSC); 2019 Oct 27-30; Auckland, New Zealand. IEEE; 2019. p. 559-66.

[98] Liu J,Bergés M,Garrett Jr JH.Detecting anomalies in longitudinal elevation of track geometry using train dynamic responses via a variational autoencoder. Proceedings of Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2019; 2019 Mar 27; Denver, CO, USA. International Society for Optics and Photonics; 2019. p. 109701B.

[99] Wang Q,He Z.Achieving predictive and proactive maintenance for high-speed railway power equipment with LSTM-RNN.IEEE Trans Ind Inf2020;16:6509-17

[100] Li D,Yan W.Acoustic emission wave classification for rail crack monitoring based on synchrosqueezed wavelet transform and multi-branch convolutional neural network.Struct Health Monit2021;20:1563-82

[101] Guo Z,Ye H.An unsupervised fault-detection method for railway turnouts.IEEE Trans Instrum Meas2020;69:8881-901

[102] Zhan Y,Yang E.Convolutional neural network for detecting railway fastener defects using a developed 3D laser system.Int J Rail Transp2021;9:424-44

[103] Li Z,Wang M,Peng F.Fiber distributed acoustic sensing using convolutional long short-term memory network: a field test on high-speed railway intrusion detection.Opt Express2020;28:2925-38

[104] Wei X,Suo D,Li Y.Multi-target defect identification for railway track line based on image processing and improved YOLOv3 model.IEEE Access2020;8:61973-88

[105] Lu J,Lei Q.SCueU-Net: efficient damage detection method for railway rail.IEEE Access2020;8:125109-20

[106] Zheng Y,Liu D,Li S.Sleeper defect detection based on improved YOLO V3 algorithm. Proceedings of 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA); 2020 Nov 9-13; Kristiansand, Norway. IEEE; 2020. p. 955-60.

[107] Zhang D,Wang Q,Wen X.Two deep learning networks for rail surface defect inspection of limited samples with line-level label.IEEE Trans Ind Inf2021;17:6731-41

[108] Chen S,Ni Y.An acoustic-homologous transfer learning approach for acoustic emission-based rail condition evaluation.Struct Health Monit2021;20:2161-81

[109] Kuzmin EV,Plotnikov PO,Bashkin VA.Application of convolutional neural networks for recognizing long structural elements of rails in eddy-current defectograms.Model anal inf sist2020;27:316-29

[110] Liu Y,Pang JHL.(, March). A YOLOv3-based deep learning application research for condition monitoring of rail thermite welded joints. Proceedings of the 2020 2nd International Conference on Image, Video and Signal Processing; 2020 Mar; New York, NY, USA. Association for Computing Machinery; 2020. p. 33-8.

[111] Aydin I,Karakose M.Defect classification based on deep features for railway tracks in sustainable transportation.Appl Soft Comput2021;111:107706

[112] Zheng D,Zheng S.A defect detection method for rail surface and fasteners based on deep convolutional neural network.Comput Intell Neurosci2021;2021:2565500 PMCID:PMC8352690

[113] Wang W,Wang W.Automated crack severity level detection and classification for ballastless track slab using deep convolutional neural network.Autom Constr2021;124:103484

[114] Chen Z,Yang K.Deep learning for the detection and recognition of rail defects in ultrasound B-scan images.Transp Res Rec2021;2675:888-901

[115] Liu J,Qiu Y,Shi B.Four discriminator cycle-consistent adversarial network for improving railway defective fastener inspection.IEEE Trans Intell Transport Syst2021;

[116] Wu Y,Qian Y,Wang Z.Hybrid deep learning architecture for rail surface segmentation and surface defect detection.Computer aided Civil Eng2022;37:227-44

[117] Wan Z.Railway tracks defects detection based on deep convolution neural networks. In: Liang Q, Wang W, Mu J, Liu X, Na Z, Cai X, editors. Artificial intelligence in China. Singapore: Springer; 2021. p. 119-29.

[118] Ye T,Zhang Y.Railway traffic object detection using differential feature fusion convolution neural network.IEEE Trans Intell Transport Syst2021;22:1375-87

[119] Chen M,Zhu S,Sun Y.Vibration-based damage detection of rail fastener using fully convolutional networks.Veh Syst Dyn2021;

[120] Tai JJ,Mehmood O.FasteNet: a fast railway fastener detector. In: Yang X, Sherratt S, Dey N, Joshi A, editors. Proceedings of Sixth International Congress on Information and Communication Technology. Singapore: Springer; 2022. p. 767-77.

[121] Guo F,Rizos D,Chen X.Automatic rail surface defects inspection based on mask R-CNN.Transp Res Rec2021;2675:655-68