Machine learning techniques for robotic and autonomous inspection of mechanical systems and civil infrastructure

Abstract

Machine learning and in particular deep learning techniques have demonstrated the most efficacy in training, learning, analyzing, and modelling large complex structured and unstructured datasets. These techniques have recently been commonly deployed in different industries to support robotic and autonomous system (RAS) requirements and applications ranging from planning and navigation to machine vision and robot manipulation in complex environments. This paper reviews the state-of-the-art with regard to RAS technologies (including unmanned marine robot systems, unmanned ground robot systems, climbing and crawler robots, unmanned aerial vehicles, and space robot systems) and their application for the inspection and monitoring of mechanical systems and civil infrastructure. We explore various types of data provided by such systems and the analytical techniques being adopted to process and analyze these data. This paper provides a brief overview of machine learning and deep learning techniques, and more importantly, a classification of the literature which have reported the deployment of such techniques for RAS-based inspection and monitoring of utility pipelines, wind turbines, aircrafts, power lines, pressure vessels, bridges, etc. Our research provides documented information on the use of advanced data-driven technologies in the analysis of critical assets and examines the main challenges to the applications of such technologies in the industry.

Keywords

Machine learning / Deep learning / Robotics and autonomous system (RAS) / Inspection / Monitoring / Mechanical systems / Civil infrastructure

References

[1] LaH.M., GucunskiN., DanaK., KeeS.-H.. Development of an autonomous bridge deck inspection robotic system. J. Field Robot., 2017, 2017: 1489

[2] WangL., ZhangZ.. Automatic detection of wind turbine blade surface cracks based on UAV-taken images. IEEE Trans. Ind. Electron., 2017, 64(9):7293-7303

[3] BernardiniS., JovanF., JiangZ., WatsonS., WeightmanA., MoradiP., RichardsonT., SadeghianR., SarehS.. A multi-robot platform for the autonomous operation and maintenance of offshore wind farms blue sky ideas track. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2020, 2020

[4] C. Stout, D. Thompson, UAV Approaches to Wind Turbine Inspection: Reducing Reliance on Rope-Access. Offshore Renewable Energy Catapult. (2019)

[5] D. Schmidt et al., Climbing robots for maintenance and inspections of vertical structures—A survey of design aspects and technologies. Robot. Auton. Syst. (2013). https://doi.org/10.1016/j.robot.2013.09.002

[6] D. Lattanzi et al., Review of Robotic Infrastructure Inspection Systems. J. Infrastruct. Syst. (2017). https://doi.org/10.1061/(ASCE)IS.1943-555X.0000353

[7] M.A.M. Yusoff et al., Development of a Remotely Operated Vehicle (ROV) for underwater inspection. Jurutera (2013)

[8] MeyrowitzA.L., et al.. Autonomous vehicles. Proceedings of the IEEE 1996, 1996

[9] F. Rubio et al., A review of mobile robots: Concepts, methods, theoretical framework, and applications. Int. J. Adv. Robot. Syst. 2019 (2019). https://doi.org/10.1177/1729881419839596

[10] D.W. Gage, A Brief History of Unmanned Ground Vehicle (UGV) Development Efforts (1995)

[11] ShenW., et al.. Proposed wall climbing robot with permanent magnetic tracks for inspecting oil tanks. IEEE International Conference Mechatronics and Automation, 2005

[12] KalraL.P., et al.. A wall climbing robot for oil tank inspection. 2006 IEEE International Conference on Robotics and Biomimetics, 2006

[13] CampbellS., et al.. Sensor technology in autonomous vehicles: a review. 2018 29th Irish Signals and Systems Conference, 2018

[14] J. Seo et al., Drone-enabled bridge inspection methodology and application. Autom. Constr. (2018). https://doi.org/10.1016/j.autcon.2018.06.006. https://www.sciencedirect.com/science/article/pii/S0926580517309755 DOI

[15] M. Shafiee et al., Unmanned Aerial Drones for Inspection of Offshore Wind Turbines: A Mission-Critical Failure Analysis. Robotics J. (2021). https://doi.org/10.3390/robotics10010026

[16] M.H. Frederiksen et al., Drones for inspection of infrastructure: Barriers, opportunities and successful uses. Center for Integrative Innovation Management (2019)

[17] M. Drones Lt, Best Commercial Drones for Beginners, Sep. 02, 2019, 2018. https://www.coptrz.com/best-commercial-drones-for-beginners/

[18] EschmannC., et al.. High-resolution multisensor infrastructure inspection with unmanned aircraft systems. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2013, 2013 https://ui.adsabs.harvard.edu/abs/2013ISPAr.XL1b.125E

[19] X.L. Ding et al., A review of structures, verification, and calibration technologies of space robotic systems for on-orbit servicing (2020). https://doi.org/10.1007/s11431-020-1737-4

[20] Flores-AbadA., et al.. A Review of Space Robotics Technologies for on-Orbit Servicing, 2014 Amsterdam Elsevier

[21] P.J. Staritz et al., Skyworker: A Robot for Assembly, Inspection and Maintenance of Large-Scale Orbital Facilities. IEEE (2001). https://doi.org/10.1109/ROBOT.2001.933271

[22] ChosetH., KortenkampD.. Path planning and control for free-flying inspection robot in space. J. Aerosp. Eng., 1999

[23] MehlingJ.S., et al.. A minimally invasive tendril robot for in-space inspection. The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, BioRob 2006, 2006 690-695

[24] NishidaS.-I., et al.. Prototype of an end-effector for a space inspection robot. Adv. Robot., 2012

[25] PedersenL., et al.. A survey of space robotics. ISAIRAS, 2003

[26] RedmonJ., et al.. You only look once: unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

[27] FanW., et al.. Mining big data: current status, and forecast to the future. 2013 Association for Computing Machinery, 2013

[28] MatturdiB., et al.. Big data security and privacy: a review. China Commun., 2014, 11(14):135-145

[29] LaneyD.. 3-D Data Management: Controlling Data Volume, Velocity and Variety, 2001

[30] BastianB.T., NJ., KumarS.K., JijiC.V.. Visual inspection and characterization of external corrosion in pipelines using deep neural network. NDT & E International Journal, 2019, 107

[31] ShihavuddinA., et al.. Wind turbine surface damage detection by deep learning aided drone inspection analysis. Energies, 2019, 12(4

[32] HassanalianM., et al.. Classifications, applications, and design challenges of drones: a review. Prog. Aerosp. Sci., 2017 https://www.sciencedirect.com/science/article/pii/S0376042116301348

[33] AlharamA., et al.. Real time AI-based pipeline inspection using drone for oil and gas industries in Bahrain. 2020 International Conference on Innovation and Intelligence for Informatics, Computing and Technologies (3ICT), 2020

[34] V. Nasteski, An overview of the supervised machine learning methods. Horizons B 4 (2017). https://doi.org/10.20544/HORIZONS.B.04.1.17.P05

[35] MaheshB.. Machine Learning Algorithms – a Review, 2019 https://doi.org/10.21275/ART20203995

[36] CarrioA., et al.. A review of deep learning methods and applications for unmanned aerial vehicles. Hindawi J. Sens., 2017

[37] M.N. Mohammed et al., Design and Development of Pipeline Inspection Robot for Crack and Corrosion Detection (2018)

[38] https://www.analyticssteps.com/blogs/how-does-k-nearest-neighbor-works-machine-learning-classification-problem

[39] HoffmannF., et al.. Benchmarking in classification and regression. WIREs Data Mining Knowl. Discov., 2019, 9

[40] A. Geron, Hands-On Machine Learning with Scikit-Learn, Keras &TensorFlow, 2nd edn. (2019). 2019

[41] GoodfellowI., et al.. Deep Learning, 2016 Cambridge MIT Press

[42] OsisanwoF.Y.. Supervised machine learning algorithms: classification and comparison. Int. J. Comput. Trends. Technol. (IJCTT), 2017, 48 3):128-138

[43] TsaiC.-F., et al.. Intrusion detection by machine learning: a review. Expert Syst. Appl., 2009, 36(10):11994-12000

[44] MatsunagaA., et al.. On the use of machine learning to predict the time and resources consumed by applications. 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, 2010 495-504

[45] JoginM., et al.. Feature extraction using Convolution Neural Networks (CNN) and deep learning. 2018 3rd IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT), 2018 2319-2323

[46] https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/

[47] https://morioh.com/p/73fce91e9846

[48] GuoY., et al.. Deep learning for visual understanding: a review. Neurocomputing, 2016, 187: 27-48

[49] K. Gopalakrishnan et al., Crack damage detection in unmanned aerial vehicle images of civil infrastructure using pre-trained deep learning model. Int. J. Traffic Transp. Eng. (IJTTE) (2017)

[50] LarochelleH., et al.. Exploring strategies for training deep neural networks. J. Mach. Learn. Res., 2009, 1: 1-40

[51] FischerA., IgelC.. An Introduction to Restricted Boltzmann Machines, 2012 Berlin Springer

[52] AgarwallaN., et al.. Deep learning using restricted Boltzmann machines. Int. J. Comput. Sci. Inf. Secur., 2016, 7(3):1552-1556

[53] HuaY., et al.. Deep belief networks and deep learning. Proceedings of 2015 International Conference on Intelligent Computing and Internet of Things, 2015 1-4

[54] https://blog.paperspace.com/faster-r-cnn-explained-object-detection/

[55] https://neurohive.io/en/popular-networks/r-cnn/

[56] ZhaoZ.-Q., et al.. Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst., 2019, 30(11):3212-3232

[57] https://blog.athelas.com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-r-cnn-34ea83205de4

[58] P.S. Bithas et al., A Survey on Machine-Learning Techniques for UAV-Based Communications. Sensors (Basel, Switzerland) 26 November 2019 (2019). https://europepmc.org/articles/PMC6929112. Accessed September 2020

[59] GirshickR., et al.. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014

[60] GirshickR.. Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, 2015

[61] K. He et al. Mask R-CNN. In ICCV, 2017

[62] J. Dai et al., R-FCN: Object Detection via Region-based Fully Convolutional Networks (2016). arXiv:1605.06409

[63] W. Liu et al., Ssd: Single shot multibox detector (2015). Preprint arXiv:1512.02325

[64] S. Ren et al., Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 2015 (2015)

[65] GrossbergS.. Recurrent neural networks. Scholarpedia, 2013, 8(2):1888

[66] J.A. Bullinaria, Recurrent neural networks. Neural Computation: Lecture 12 (2013)

[67] HochreiterS., SchmidhuberJ.. Long short-term memory. Neural Comput., 1997, 9(8):1735-1780

[68] I.J. Goodfellow et al., Generative Adversarial Networks (2014). arXiv, stat.ML

[69] M. Mirza, S. Osindero, Conditional Generative Adversarial Nets (2014)

[70] L. Noriega, Multilayer perceptron tutorial. School of Computing. Staffordshire University (2005)

[71] TaudH., MasJ.. Multilayer perceptron (MLP). Geomatic Approaches for Modeling Land Change Scenarios, 2018 Cham Springer

[72] G. Alain, Y. Bengio, What Regularized Auto-Encoders Learn from the Data Generating Distribution (2014)

[73] LauzonF.Q.. An introduction to deep learning. 2012 11th International Conference on Information Science, Signal Processing and Their Applications (ISSPA), 2012 1438-1439

[74] BaldiP.. Autoencoders, unsupervised learning, and deep architectures. Proceedings of ICML Workshop on Unsupervised and Transfer Learning, 2012

[75] LeQ.V.. A tutorial on deep learning part 2: autoencoders, convolutional neural networks and recurrent neural networks. Google Brain, 2015, 20: 1-20

[76] AgarwalA., MotwaniA.. An Overview of Convolutional and AutoEncoder Deep Learning Algorithm, 2016

[77] CoadouY.. Boosted decision trees and applications. EPJ Web Conf., 2013, 55

[78] Beauxis-AussaletE., et al.. Visualization of confusion matrix for non-expert users. IEEE Conference on Visual Analytics Science and Technology (VAST) - Poster Proceedings, 2014

[79] ShobhaG., et al.. Machine learning. Handbook of Statistics, 2018 Amsterdam Elsevier 197-228 https://www.sciencedirect.com/science/article/pii/S0169716118300191. ISSN 0169-7161. ISBN 9780444640420

[80] KulkarniA., et al.. Foundations of data imbalance and solutions for a data democracy. Data Democracy, 2020 San Diego Academic Press 83-106 ISBN 9780128183663

[81] https://towardsdatascience.com/map-mean-average-precision-might-confuse-you-5956f1bfa9e2

[82] MohamedN., et al.. Real-time big data analytics: applications and challenges. Proc. Int. Conf. High Perform. Comput. Simulation, 2014 305-310

[83] J. Franko et al., Design of a multi-robot system for wind turbine maintenance. Energies (2020)

[84] BrandoliB., et al.. Aircraft fuselage corrosion detection using artificial intelligence. Sensors, 2021, 2021(21

[85] T. Malekzadeh et al., Aircraft Fuselage Defect Detection using Deep Neural Networks (2017). arXiv:1712.09213

[86] MirandaJ., et al.. Machine learning approaches for defect classification on aircraft fuselage images aquired by an UAV. Fourteenth International Conference on Quality Control by Artificial Vision, 2019 1117208

[87] JalilB., et al.. Fault detection in power equipment via an unmanned aerial system using multi modal data. Sensors, 2019, 2019 19

[88] TitovE., et al.. The deep learning based power line defect detection system built on data collected by the cablewalker drone. 2019 International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON), 2019 0700-0704

[89] OrtizA., et al.. First steps towards a roboticized visual inspection system for vessels. 2010 IEEE 15th Conference on Emerging Technologies & Factory Automation (ETFA 2010), 2010 1-6

[90] Bonnin-PascualF., et al.. Semi-autonomous visual inspection of vessels assisted by an unmanned micro aerial vehicle. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012 3955-3961

[91] KawabataS., et al.. Autonomous flight drone with depth camera for inspection task of infra structure. Proceedings of the International MultiConference of Engineers and Computer Scientists, 2018

[92] ChaY.-J., et al.. Deep learning-based crack damage detection using convolutional neural networks. Comput.-Aided Civ. Infrastruct. Eng., 2017, 32: 361-378

[93] KarimM.M., et al.. Modeling and simulation of a robotic bridge inspection system. Procedia Computer Science, 2020 177-185 https://www.sciencedirect.com/science/article/pii/S1877050920304154. ISSN 1877-0509

[94] ManzoorM.A., et al.. Vehicle make and model classification system using bag of SIFT features. 2017 IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC), 2017 1-5

[95] P. Rakshata et al., Car damage detection and analysis using deep learning algorithm for automotive. Int. J. Sci. Technol. Res. 5(6) (2019). Nov-Dec-2019, ISSN (Online): 2395-566X

[96] ZhangQ., et al.. Vehicle-damage-detection segmentation algorithm based on improved mask RCNN. IEEE Access, 2020, 8: 6997-7004

[97] BandiH., et al.. Assessing car damage with convolutional neural networks. 2021 International Conference on Communication Information and Computing Technology (ICCICT), 2021 1-5

[98] C. Giovany Pachón-Suescún et al., Scratch Detection in Cars Using a Convolutional Neural Network by Means of Transfer Learning. IJAER (2018) 16 Nov 2018

[99] PatilK., et al.. Deep learning based car damage classification. 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), 2017 50-54

[100] R. Ali et al., Subsurface damage detection of a steel bridge using deep learning and uncooled micro-bolometer. Constr. Build. Mater. 226 (2019). https://doi.org/10.1016/j.conbuildmat.2019.07.293. 2019, 376-387, ISSN 0950-0618. https://www.sciencedirect.com/science/article/pii/S0950061819319671

[101] LiuY., et al.. The method of insulator recognition based on deep learning. Proceedings of the 2016 4th International Conference on Applied Robotics for the Power Industry (CARPI), 2016 1-5

[102] ZhaoZ., et al.. Multi-patch deep features for power line insulator status classification from aerial images. 2016 International Joint Conference on Neural Networks (IJCNN), 2016, 2016 3187-3194

[103] OrtizA., et al.. Vision-based corrosion detection assisted by a micro-aerial vehicle in a vessel inspection application. Sensors, 2016, 2016 16

[104] V.N. Nguyen et al., Automatic autonomous vision-based power line inspection: A review of current status and the potential role of deep learning. Int. J. Electr. Power Energy Syst. 2018 (2018)

[105] Faghih-RoohiS., et al.. Deep convolutional neural networks for detection of rail surface defects. Neural Networks (IJCNN), 2016 International Joint Conference on, 2016, 2016 2584-2589

[106] TongZ.-Q., et al.. Recent advances in small object detection based on deep learning: a review. Image Vis. Comput., 2020, 97

[107] LinT.-Y., et al.. Microsoft Coco: Common Objects in Context, 2014 Cham Springer

[108] LiuY., et al.. A survey and performance evaluation of deep learning methods for small object detection. Expert Syst. Appl., 2021, 172

[109] NguyenN.-D., et al.. An evaluation of deep learning methods for small object detection. J. Electr. Comput. Eng., 2020, 2020

[110] ChenyiC., et al.. R-CNN for small object detection. Asian Conference on Computer Vision, 2016 Cham Springer

[111] ZhaoZ.-Q., et al.. Object detection with deep learning: a review. IEEE Trans. Neural Netw. Learn. Syst., 2019, 30 11):3212-3232

[112] ShenY., et al.. Design alternative representations of confusion matrices to support non-expert public understanding of algorithm performance. Proc. ACM Hum. Comput. Interact., 2020, 4(CSCW2

[113] R.K. Rai et al., Intricacies of Unstructured Data. EAI Endorsed Transactions on Scalable Information Systems 4(14) (2017). https://doi.org/10.4108/eai.25-9-2017.153151

[114] GandomiA., et al.. Beyond the hype: big data concepts, methods, and analytics. Int. J. Inf. Manag., 2015, 35 2):137-144

[115] AcharjyaD.P., et al.. A survey on big data analytics: challenges, open research issues and tools. Int. J. Adv. Comput. Sci. Appl., 2016, 7: 511-518

[116] OussousA., et al.. Big data technologies: a survey. J. King Saud Univ, Comput. Inf. Sci., 2018, 30: 431-448

[117] JinX., et al.. Significance andchallenges of big data research. Big Data Res., 2015, 2(2):59-64

[118] NajafabadiM.M., et al.. Deep learning applications and challenges in big data analytics. Big Data, 2015, 2(1):1-21

[119] PanellaI.. Artificial intelligence methodologies applicable to support the decision-making capability on board unmanned aerial vehicles. ECSIS Symposium on Bio-Inspired Learning and Intelligent Systems for Security, 2008 111-118

[120] OnoM., et al.. MAARS: machine learning-based analytics for automated rover systems. Proc. IEEE Aerosp. Conf, 2020 1-17

[121] M. Hillebrand et al., A design methodology for deep reinforcement learning in autonomous systems. Procedia Manufacturing 52 (2020). https://doi.org/10.1016/j.promfg.2020.11.044. https://www.sciencedirect.com/science/article/pii/S2351978920321879

[122] ContrerasS., et al.. Using deep learning for exploration and recognition of objects based on images. 2016 XIII Latin American Robotics Symposium and IV Brazilian Robotics Symposium (LARS/SBR), 2016 1-6

[123] ChenW., et al.. Door recognition and deep learning algorithm for visual based robot navigation. 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), 2014 1793-1798

Funding

ESPRC(2466663)