Advances in computer vision for comprehensive railway engineering: from track inspection to rolling stock and safety monitoring

Abstract

Railways form a vital part of global transportation networks, supporting economic development and sustainability through the efficient movement of freight and passengers. Maintaining the safety and reliability of these systems requires consistent inspection and monitoring of both infrastructure and rolling stock. Traditional inspection practices, while established, remain labour-intensive, time-consuming, and prone to human error, which can undermine operational reliability. Computer vision has become a key technology for automating these processes by enabling accurate visual assessment of tracks, components, and operational environments. Developments in deep learning have significantly strengthened these capabilities by improving defect detection accuracy, enhancing robustness under varied environmental conditions, and supporting real-time operation. More recently, AI-driven inspection frameworks that incorporate multi-sensor fusion, edge computing, and transformer-based architectures have pushed railway monitoring towards predictive, scalable, and increasingly autonomous maintenance solutions. This review provides a comprehensive and structured synthesis of research spanning track inspection, rolling stock monitoring, and passenger and operational safety. It evaluates recent methodological advances, summarises the strengths and limitations of current approaches, and identifies ongoing challenges related to data availability, computational constraints, and deployment in dynamic outdoor settings. The evidence indicates that track inspection technologies are closest to large-scale practical adoption, while rolling stock and safety monitoring systems are developing rapidly but still require further refinement for widespread deployment. By offering an integrated assessment of the current technological landscape and outlining opportunities for future research, this review supports researchers, industry practitioners, and policymakers in progressing towards safer, more efficient, and more sustainable railway systems enabled by modern computer vision and AI techniques.

Keywords

Computer vision / Deep learning / AI-driven systems / Autonomous railway inspection / Real-time monitoring / Predictive maintenance / IoT-based monitoring / Fault detection / Railway engineering / Safety monitoring

References

[1] Hale A. Railway safety management: the challenge of the new millennium. Saf Sci Monit, 2000, 1(1): 1-15

[2] Peira G, Lo Giudice A, Miraglia S. Railway and tourism: a systematic literature review. Tourism Hosp, 2022, 3(1): 69-79

[3] D’Alfonso T, Jiang C, Bracaglia V. Air transport and high-speed rail competition: environmental implications and mitigation strategies. Transport Res Part A Policy Pract, 2016, 92: 261-276

[4] Dinmohammadi F. A risk-based modelling approach to maintenance optimization of railway rolling stock: a case study of pantograph system. J Qual Maint Eng, 2019, 25(2): 272-293

[5] He L, Liang Q, Fang S. Challenges and innovative solutions in urban rail transit network operations and management: China’s guangzhou metro experience. Urban Rail Transit, 2016, 2: 33-45

[6] Singh P, Dulebenets MA, Pasha J et al (2021) Deployment of autonomous trains in rail transportation: current trends and existing challenges. IEEE Access 9:91427–91461

[7] Jing G, Qin X, Wang H et al (2022) Developments, challenges, and perspectives of railway inspection robots. Autom Constr 138:104242

[8] Chen Y (2013) Improving railway safety risk assessment study. Dissertation, University of Birmingham

[9] Belkhade A, Kathale S. Automatic vision based inspection of railway track: a review. Int J Res Eng Technol, 2014, 3(2): 12-15

[10] Kovacevic M, Gavin K, Oslakovic IS et al (2016) A new methodology for assessment of railway infrastructure condition. Transportation research procedia 14:1930–1939

[11] Remennikov AM, Kaewunruen S. A review of loading conditions for railway track structures due to train and track vertical interaction. Struct Control Health Monit Off J Int Assoc Struct Control Monit Eur Assoc Control Struct, 2008, 15(2): 207-234

[12] Wang Z, Wang S (2015) Research of method for detection of rail fastener defects based on machine vision. In: 2015 4th International conference on mechatronics, materials, chemistry and computer engineering, Atlantis Press

[13] Zhao K, Luo L, Ren Z et al (2020) A surface defect detection system for railway track based on machine vision. Journal of Physics: Conference Series 1678:012002

[14] Hu Y, Mao JX, Mao JP. Research on rail surface defect inspection system. Appl Mech Mater, 2012, 229: 1389-1393

[15] Li Y, Trinh H, Haas N, Otto C, Pankanti S. Rail component detection, optimization, and assessment for automatic rail track inspection. IEEE Trans Intell Transp Syst, 2013, 15(2): 760-770

[16] Loktev DA, Loktev AA (2018) Diagnostics of external defects of railway infrastructure by analysis of its images. In: 2018 Global smart industry conference (GloSIC), Chelyabinsk, pp 1–7

[17] Hua J, Wang Z, Han H et al (2024) A novel robotic-assisted deep learning-enabled computer vision approach for nondestructive diagnosis of railway bolt faults. Meas Sci Technol 35(9):096118

[18] Alif MAR, Hussain M, Tucker G, Iwnicki S. Boltvision: a comparative analysis of cnn, cct, and vit in achieving high accuracy for missing bolt classification in train components. Machines, 2024, 12(2): 93

[19] Alif MAR, Hussain M. Lightweight convolutional network with integrated attention mechanism for missing bolt detection in railways. Metrology, 2024, 4(2): 254-278

[20] Zhuang L, Qi H, Zhang Z. The automatic rail surface multi-flaw identification based on a deep learning powered framework. IEEE Trans Intell Transp Syst, 2021, 23(8): 12133-12143

[21] Kou L, Sysyn M, Fischer S, Liu J, Nabochenko O. Optical rail surface crack detection method based on semantic segmentation replacement for magnetic particle inspection. Sensors, 2022, 22(21): 8214

[22] Alif MAR. State-of-the-art bangla handwritten character recognition using a modified resnet-34 architecture. Int J Innov Sci Res Technol, 2024, 9: 438-448

[23] Alif MAR, Ahmed S, Hasan MA (2017) Isolated bangla handwritten character recognition with convolutional neural network. In: 20th International conference of computer and information technology (ICCIT), Dhaka, pp 1–6

[24] Devarapalli RL, Yalamanchili S, Sri RK (2023) Broken rail track detection system using transfer learning. In: 2023 2nd International conference on applied artificial intelligence and computing (ICAAIC), Salem, pp 276–279

[25] Luo H, Cai L, Li C. Rail surface defect detection based on an improved yolov5s. Appl Sci, 2023, 13(12): 7330

[26] Redmon J (2015) You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, pp 779–788

[27] Alif MAR. Attention-based automated pallet racking damage detection. Int J Innov Sci Res Technol, 2024, 9: 728-740

[28] Kocbek S, Gabrys B (2019) Automated machine learning techniques in prognostics of railway track defects. In: 2019 International conference on data mining workshops (ICDMW. Beijing), pp 777–784

[29] Li X, Liu Q, Liu T et al (2021) Research on yolo model and its application in fault status recognition of freight trains. In: Advances in artificial intelligence and security: 7th international conference, Dublin, pp 144–156

[30] Kafetzis D, Fourfouris I, Argyropoulos S et al (2020) UAV-assisted aerial survey of railways using deep learning. In: 2020 International Conference on Unmanned Aircraft Systems (ICUAS). Athens, pp. 1491–1500

[31] Sun J, Xie Y, Cheng X. A fast bolt-loosening detection method of running train’s key components based on binocular vision. IEEE Access, 2019, 7: 32227-32239

[32] Binder M, Mezhuyev V, Tschandl M. Predictive maintenance for railway domain: a systematic literature review. IEEE Eng Manage Rev, 2023, 51(2): 120-140

[33] Pahwa RS, Chao J, Paul J et al (2019) Faultnet: faulty rail-valves detection using deep learning and computer vision. In: 2019 IEEE intelligent transportation systems conference (ITSC), Auckland, pp 559–566

[34] Lin Y-W, Hsieh C-C, Huang W-H et al (2019) Railway track fasteners fault detection using deep learning. In: 2019 IEEE Eurasia conference on IOT, communication and engineering (ECICE). Yunlin, pp 187–190

[35] Karakose M, Yaman O, Baygin M, Murat K, Akin E. A new computer vision based method for rail track detection and fault diagnosis in railways. Int J Mech Eng Robot Res, 2017, 6(1): 22

[36] Karakose M, Yaman O, Murat K, Akin E. A new approach for condition monitoring and detection of rail components and rail track in railway. Int J Comput Intell Syst, 2018, 11(1): 830-845

[37] Xia B, Cao J, Zhang X, Peng Y. Automatic concrete sleeper crack detection using a one-stage detector. Int J Intell Robot Appl, 2020, 4(3): 319-327

[38] Alawad H, Kaewunruen S, An M. A deep learning approach towards railway safety risk assessment. IEEE Access, 2020, 8: 102811-102832

[39] Zaman A, Ren B, Liu X. Artificial intelligence-aided automated detection of railroad trespassing. Transp Res Rec, 2019, 2673(7): 25-37

[40] Zhang Z, Trivedi C, Liu X. Automated detection of grade-crossing-trespassing near misses based on computer vision analysis of surveillance video data. Saf Sci, 2018, 110: 276-285

[41] Guo F, Qian Y, Rizos D, Suo Z, Chen X. Automatic rail surface defects inspection based on mask r-cnn. Transp Res Rec, 2021, 2675(11): 655-668

[42] Zhang P, Lei W, Zhang W et al (2024) A computer vision based warning and behavior analysis system for large passenger flow in stations. In: 2024 8th International conference on management engineering, software engineering and service sciences (ICMSS). Wuhan, pp 126–132

[43] Huang H, Moaveni M, Schmidt S, Tutumluer E, Hart JM. Evaluation of railway ballast permeability using machine vision–based degradation analysis. Transp Res Rec, 2018, 2672(10): 62-73

[44] Zhangyu W, Guizhen Y, Xinkai W, Haoran L, Da L. A camera and lidar data fusion method for railway object detection. IEEE Sens J, 2021, 21(12): 13442-13454

[45] Marino F, Distante A, Mazzeo PL, Stella E. A real-time visual inspection system for railway maintenance: automatic hexagonal-headed bolts detection. IEEE Trans Syst Man Cybern Part C (Applications and Reviews), 2007, 37(3): 418-428

[46] Shaikh MZ, Ahmed Z, Baro EN, Hussain S, Milanova M. Deep learning based identification and tracking of railway bogie parts. Alex Eng J, 2024, 107: 533-546

[47] Shang Z, Li L, Zheng S, Mao Y, Shi R. Fiq: a fastener inspection and quantization method based on mask frcn. Appl Sci, 2024, 14(12): 5267

[48] James A, Wang J, Yang X et al (2018) Tracknet—a deep learning based fault detection for railway track inspection. In: 2018 International conference on intelligent rail transportation (ICIRT). Singapore, pp 1–5

[49] Zhuang L, Qi H, Wang T, Zhang Z. A deep-learning-powered near-real-time detection of railway track major components: a two-stage computer-vision-based method. IEEE Internet Things J, 2022, 9(19): 18806-18816

[50] Han K, Wang Y, Chen H, Chen X, Guo J, Liu Z, Tang Y, Xiao A, Xu C, Xu Y, et al. . A survey on vision transformer. IEEE Trans Pattern Anal Mach Intell, 2022, 45(1): 87-110

[51] Ou Y, Luo J, Li B, He B. A classification model of railway fasteners based on computer vision. Neural Comput Appl, 2019, 31: 9307-9319

[52] Martin D, Heinzel S, Bischhoffshausen JK et al (2021) Deep learning strategies for industrial surface defect detection systems. arXiv preprint arXiv:2109.11304

[53] Ozdemir R, Koc M. On the enhancement of semi-supervised deep learning-based railway defect detection using pseudo-labels. Expert Syst Appl, 2024, 251 124105

[54] Carlini N, Hayes J et al (2023) Extracting training data from diffusion models. In: 32nd USENIX security symposium (USENIX Security 23). Anaheim, pp 5253–5270

[55] Zhang D, Xu P, Tian Y, Zhong C, Zhang X. Ballasted track behaviour induced by absent sleeper support and its detection based on a convolutional neural network using track data. Urban Rail Transit, 2023, 9(2): 92-109

[56] Siahkouhi M, Wang J, Han X, Aela P, Ni Y-Q, Jing G. Railway ballast track hanging sleeper defect detection using a smart cnt self-sensing concrete railway sleeper. Constr Build Mater, 2023, 399 132487

[57] Franca AS, Vassallo RF. A method of classifying railway sleepers and surface defects in real environment. IEEE Sens J, 2020, 21(10): 11301-11309

[58] Lesiak P, Bojarczak P, Sokołowski A (2023) Algorithm for the extraction of selected rail track ballast degradation using machine vision. Transp Probl 18

[59] Guerrieri M, Parla G, Celauro C. Digital image analysis technique for measuring railway track defects and ballast gradation. Measurement, 2018, 113: 137-147

[60] Yella S, Dougherty M, Gupta NK. Condition monitoring of wooden railway sleepers. Transp Res Part C Emerg Technol, 2009, 17(1): 38-55

[61] Shi D, Šabanovič E, Rizzetto L, Skrickij V, Oliverio R, Kaviani N, Ye Y, Bureika G, Ricci S, Hecht M. Deep learning based virtual point tracking for real-time target-less dynamic displacement measurement in railway applications. Mech Syst Signal Process, 2022, 166 108482

[62] Zhang Q, Li Y, Zhang J, Luo Y. Insar technique applied to the monitoring of the qinghai–tibet railway. Nat Hazard, 2019, 19(10): 2229-2240

[63] Banić M, Miltenović A, Pavlović M, Ćirić I. Intelligent machine vision based railway infrastructure inspection and monitoring using uav. Facta Universitatis Series Mech Eng, 2019, 17(3): 357-364

[64] Wen T, Liu J, Roberts C (2025) A high-precision deep Kalman filter design for nonlinear train operation state estimation. IEEE Trans Veh Technol 74(7):10393–10406

[65] Liu L, Zhou F, He Y. Vision-based fault inspection of small mechanical components for train safety. IET Intel Transp Syst, 2016, 10(2): 130-139

[66] Ye T, Zhang X, Zhang Y, Liu J. Railway traffic object detection using differential feature fusion convolution neural network. IEEE Trans Intell Transp Syst, 2020, 22(3): 1375-1387

[67] Yu M, Yang P, Wei S (2018) Railway obstacle detection algorithm using neural network. AIP Conference proceedings 1967(1):040017

[68] Gorane YM, Joshi RD (2024) Railway accident reduction by passenger detection using machine learning techniques. In: IEEE 9th international conference for convergence in technology (I2CT). Pune, pp 1–6

[69] Alawad H, Kaewunruen S, An M. Learning from accidents: machine learning for safety at railway stations. IEEE Access, 2019, 8: 633-648

[70] Wu Y, Jin W, Li Y, Sun Z, Ren J. Detecting unexpected faults of high-speed train bogie based on bayesian deep learning. IEEE Trans Veh Technol, 2020, 70(1): 158-172

[71] Kou L, Qin Y, Zhao X. A multi-dimension end-to-end cnn model for rotating devices fault diagnosis on high-speed train bogie. IEEE Trans Veh Technol, 2019, 69(3): 2513-2524

[72] Chen L, Qin N, Dai X, Huang D. Fault diagnosis of high-speed train bogie based on capsule network. IEEE Trans Instrum Meas, 2020, 69(9): 6203-6211

[73] Dumitriu M. Fault detection of damper in railway vehicle suspension based on the cross-correlation analysis of bogie accelerations. Mech Ind, 2019, 20(1): 102

[74] Prasad CR, Kishore P. Performance of active contour models in train rolling stock part segmentation on high-speed video data. Cogent Eng, 2017, 4(1): 1279367

[75] Kishore P, Prasad CR. Train rolling stock intelligent monitoring with computer vision. Mater Today Proc, 2017, 4(2): 1730-1739

[76] Kishore P, Prasad CR. Computer vision based train rolling stock examination. Optik, 2017, 132: 427-444

[77] Singh S, Kumar R. Evaluation of human error probability of disc brake unit assembly and wheel set maintenance of railway bogie. Procedia Manuf, 2015, 3: 3041-3048

[78] Zhou J, Huo L. Computer vision-based detection for delayed fracture of bolts in steel bridges. J Sensors, 2021, 2021(1): 8325398

[79] Junfeng A, Jiqiang L, Mengmeng L, Gang L, Huasheng S. Image change combined with cnn power subway vent valve state monitoring. SIViP, 2024, 18(3): 2151-2166

[80] Zhou Z, Hu Y, Deng X et al (2021) Fault detection of train height valve based on Nanodet-Resnet101. In: 36th Youth academic annual conference of Chinese association of automation (YAC). Nanchang, pp 709–714

[81] Bhorge S, Shelke V, Shintre K et al (2024) Intelligent thermal management system. In: International conference on artificial intelligence and quantum computation-based sensor application (ICAIQSA). Nagpur, pp 1–6

[82] Liu L, Zhou F, He Y. Automated status inspection of fastening bolts on freight trains using a machine vision approach. Proc Inst Mech Eng Part F J Rail Rapid Transit, 2016, 230(7): 1629-1641

[83] Ozdemir R, Koc M. On the enhancement of semi-supervised deep learning-based railway defect detection using pseudo-labels. Expert Syst Appl, 2024, 251 124105

[84] Liu T, Ye W. A semi-supervised learning method for surface defect classification of magnetic tiles. Mach Vis Appl, 2022, 33(2): 35

[85] Shi C, Wang K, Zhang G, Li Z, Zhu C. Efficient and accurate semi-supervised semantic segmentation for industrial surface defects. Sci Rep, 2024, 14(1): 21874

[86] Min Y, Li Y. Self-supervised railway surface defect detection with defect removal variational autoencoders. Energies, 2022, 15(10): 3592

[87] Li Z, Yan Y, Wang X, Ge Y, Meng L. A survey of deep learning for industrial visual anomaly detection. Artif Intell Rev, 2025, 58(9): 279

[88] Shah RA, Urmonov O, Kim H (2025) Self-supervised image anomaly detection through diverse pseudo anomaly insertion. IEEE Access 13:115712–115734

[89] Zheng Z, Qi H, Zhuang L, Zhang Z. Automated rail surface crack analytics using deep data-driven models and transfer learning. Sustain Cities Soc, 2021, 70 102898

[90] Usama M, Qadir J, Raza A, Arif H, Yau K-LA, Elkhatib Y, Hussain A, Al-Fuqaha A. Unsupervised machine learning for networking: techniques, applications and research challenges. IEEE access, 2019, 7: 65579-65615

[91] Andersen RE, Brogaard RY, Boukas E (2024) Remote inspection techniques: a review of autonomous robotic inspection for marine vessels. IEEE Trans Field Robot 2:1–20

[92] Tucker G, Hall A (2014) Breaking down the barriers to more cross-industry remote condition monitoring (RCM). In: 6th IET Conference on railway condition monitoring (RCM 2014). Birmingham, pp 1–6

[93] Saini N, Yadav AL, Rahman A (2024) Cloud based predictive maintenance system. In: 11th international conference on reliability, infocom technologies and optimization (Trends and Future Directions). Noida, pp 1–5

[94] Siddiqui HUR, Saleem AA, Raza MA, Zafar K, Munir K, Dudley S. Iot based railway track faults detection and localization using acoustic analysis. IEEE Access, 2022, 10: 106520-106533

[95] Chellaswamy C, Balaji L, Vanathi A et al (2017) IoT based rail track health monitoring and information system. In: 2017 International conference on microelectronic devices, circuits and systems (ICMDCS). Vellore, pp 1–6

[96] Wang Y, Kuang B, Durazo I et al (2024) 3D reconstruction of rail tracks based on fusion of RGB and infrared sensors. In: 29th international conference on automation and computing (ICAC). Sunderland, pp 1–6

[97] Singh P, Elmi Z, Meriga VK, Pasha J, Dulebenets MA. Internet of things for sustainable railway transportation: past, present, and future. Cleaner Logist Supply Chain, 2022, 4 100065

[98] Chen H, Jiang B, Chen W, Li Z. Edge computing-aided framework of fault detection for traction control systems in high-speed trains. IEEE Trans Veh Technol, 2019, 69(2): 1309-1318

[99] Ćirić I, Banić M, Simonović M, Miltenović A, Stamenković D, Nikolić V. Towards machine vision based railway assets predictive maintenance. Facta Univ Series Autom Control Robot, 2020, 19(2): 125-136

[100] Yu S-D (2021) Research on cloud computing in the key technologies of railway intelligent operation and maintenance sharing platform. Journal of physics: conference series 1800: pp 012010

[101] Gbadamosi A-Q, Oyedele LO, Delgado JMD, Kusimo H, Akanbi L, Olawale O, Muhammed-Yakubu N. Iot for predictive assets monitoring and maintenance: An implementation strategy for the uk rail industry. Autom Constr, 2021, 122 103486

[102] Fu Y (2025) Efficient real-time railway fault detection using fpga-based edge ai system. Dissertation, The University of Auckland

[103] Liu Z, Li F, Huang B, Zhang G. Real-time and accurate rail wear measurement method and experimental analysis. J Opt Soc Am A, 2014, 31(8): 1721-1729

[104] Louadah H, Papadakis E, McCluskey TL, Tucker G. Supporting the management of rolling stock maintenance with an ontology-based virtual depot. Appl Sci, 2024, 14(3): 1220

RIGHTS & PERMISSIONS

The Author(s)