Deep Reinforcement Learning Approach for X-rudder AUVs Fault Diagnosis Based on Deep Q-network

Abstract

The rudder mechanism of the X-rudder autonomous underwater cehicle (AUV) is relatively complex, and fault diagnosis capability is an important guarantee for its task execution in complex underwater environments. However, traditional fault diagnosis methods currently rely on prior knowledge and expert experience, and lack accuracy. In order to improve the autonomy and accuracy of fault diagnosis methods, and overcome the shortcomings of traditional algorithms, this paper proposes an X-steering AUV fault diagnosis model based on the deep reinforcement learning deep Q network (DQN) algorithm, which can learn the relationship between state data and fault types, map raw residual data to corresponding fault patterns, and achieve end-to-end mapping. In addition, to solve the problem of few X-steering fault sample data, Dropout technology is introduced during the model training phase to improve the performance of the DQN algorithm. Experimental results show that the proposed model has improved the convergence speed and comprehensive performance indicators compared to the unimproved DQN algorithm, with precision, recall, F _1−score, and accuracy reaching up to 100%, 98.07%, 99.02%, and 98.50% respectively, and the model’s accuracy is higher than other machine learning algorithms like back propagation, support vector machine.

References

[] Alessandri A. Fault diagnosis for nonlinear systems using a bank of neural estimators Computers in Industry, 2003, 52(3): 271-289.

[] Alex Gong CS, Simon Su CH, Tseng KH. Implementation of Machine Learning for Fault Classification on Vehicle Power Transmission System IEEE Sensors Journal, 2020, 20(24): 15163-15176.

[] Antonelli G, Caccavale F, Sansone C, Villani L. Fault diagnosis for AUVs using support vector machines IEEE International Conference on Robotics and Automation 4486–4491, 2004

[] Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA. Deep Reinforcement Learning: A Brief Survey IEEE Signal Processing Magazine, 2017, 34(6): 26-38.

[] Chang ZH, Jia KW, Han T, Wei YM. Towards more reliable photovoltaic energy conversion systems: A weakly-supervised learning perspective on anomaly detection Energy Conversion and Management, 2024, 316: 118845.

[] Chen Y, Mabu S, Hirasawa K, Hu J. Enhancement of trading rules on stock markets using genetic network programming with Sarsa learning SICE Annual Conference, 2007, 2007: 2700-2707

[] Ding Y, Ma L, Ma J, Suo M, Tao L, Cheng Y, Lu C. Intelligent fault diagnosis for rotating machinery using deep Q-network based health state classification: A deep reinforcement learning approach Advanced Engineering Informatics, 2019, 42: 100977.

[] Fang M, Li H, Zhang X. A Heuristic Reinforcement Learning Based on State Backtracking Method 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, 2012 673-678.

[] Ferguson J, Pope A. Explorer-a modular AUV for commercial site survey Proceedings of the 2000 International Symposium on Underwater Technology (Cat. No. 00EX418), 2000 129-132.

[] Frank PM. Fault Diagnosis in Dynamic Systems Using Analytical and Knowledge-based Redundancy A Survey and Some New Results Automatica, 1990, 26(3): 459-474.

[] Geng H, Liu H, Wang B, Sun F Cao J, Cambria E, Lendasse A, Miche Y, Vong C M. Reinforcement Extreme Learning Machine for Mobile Robot Navigation Proceedings of ELM-2016, 2018 61-73. Vol. 9)

[] Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR Improving neural networks by preventing co-adaptation of feature detectors, 2012((arXiv: 1207.0580))

[] Huang Y, Li Y, Yu JC, Li S, Feng XS. State-of-the-Art and Development Trends of AUV Intelligence Robot, 2020, 42(02): 215-231((in Chinese))

[] Ji D, Yao X, Li S, Tang Y, Tian Y. Model-free fault diagnosis for autonomous underwater vehicles using sequence Convolutional Neural Network Ocean Engineering, 2021, 232: 108874.

[] Jiang Y, Yin S. Recent Advances in Key-Performance-Indicator Oriented Prognosis and Diagnosis With a MATLAB Toolbox: DB-KIT IEEE Transactions on Industrial Informatics, 2019, 15(5): 2849-2858.

[] Jiang Y, Yin S, Kaynak O. Data-driven monitoring and Safety Control of Industrial Cyber-Physical Systems: Basics and Beyond IEEE Access, 2018, 6: 47374-47384.

[] Liu F, Xu D. Fault Localization and Fault-Tolerant Control for rudders of AUVs 2016 35th Chinese Control Conference (CCC), 2016 6537-6541.

[] Miskovic N, Barisic M. Fault Detection and Localization on Underwater Vehicle Propulsion Systems Using Principal Component Analysis Proceedings of the IEEE International Symposium on Industrial Electronics, 2005 1721-1728

[] Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M, Rusu AA, Veness J, Bellemare MG, Krizhevsky A, Hinton G, Hassabis D Playing atari with deep reinforcement learning, 2013(arXiv preprint arXiv: 1312.5602)

[] Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D. Human-level control through deep reinforcement learning Nature, 2015, 518(7540): 529-533.

[] Pan Y, Zheng Z, Fu D. Bayesian-based water leakage detection with a novel multisensor fusion method in a deep manned submersible Applied Ocean Research, 2021, 106: 102459.

[] Prestero T Verification of a six-degree of freedom simulation model for the REMUS autonomous underwater vehicle, 2001.

[] Shao Z, Hua YX, Shi JY. Research on Fault Diagnosis Method for Rudder Surface Based on Multiple Models Flight Dynamics, 2020, 38(3): 24-27((in Chinese))

[] Sun Y, Ran X, Li Y, Zhang G, Zhang Y. Thruster fault diagnosis method based on Gaussian particle filter for autonomous underwater vehicles International Journal of Naval Architecture and Ocean Engineering, 2016, 8(3): 243-251.

[] Sun Y, Wang Z, Zhang G. Fault Diagnosis Method of Autonomous Underwater Vehicle Based on Deep Learning IOP Conference Series: Materials Science and Engineering, 2019, 470: 012035.

[] Tun W, Wong J W, Ling SH. Hybrid Random Forest and Support Vector Machine Modeling for HVAC Fault Detection and Diagnosis Sensors, 2021, 21(24): 8163.

[] Wang F, Wan L, Su Y, Xu Y. AUV modeling and motion control strategy design Journal of Marine Science and Application, 2010, 9(4): 379-385.

[] Wang LR, Gan Y, Xu YR, Wan L. Sliding-mode observers used in thruster fault diagnosis of an autonomous underwater vehicle Journal of Harbin Engineering University, 2005, 26(4): 425-429

[] Wang X, Fang X. A multi-agent reinforcement learning algorithm with the action preference selection strategy for massive target cooperative search mission planning Expert Systems with Applications, 2023, 231: 120643.

[] Wang YJ, Zhang MJ, Che ZZ. Qualitative diagnostic model for underwater robot propeller fault Proceedings of the Seventh National Conference on Fault Diagnosis and Safety of Technical Processes, 2011

[] Wang YJ, Zhang MJ, Wu J. Research of the fault diagnosis method for the thruster of AUV based on information fusion Third International Conference on Agent Computing, 2007

[] Watkins CJ, Dayan P. Q-learning Machine learning, 1992, 8: 279-292.

[] Wen L, Wang Y, Li X. A new automatic convolutional neural network based on deep reinforcement learning for fault diagnosis Frontiers of Mechanical Engineering, 2022, 17(2): 17.

[] Wu X, Xiong W. kNN Fault Detection Based on Multi-block Information Extraction and Mahalanobis Distance Information and Control, 2021, 50(3): 287-296

[] Xing B, Wang X, Liu Z. The Wide-Area Coverage Path Planning Strategy for Deep-Sea Mining Vehicle Cluster Based on Deep Reinforcement Learning Journal of Marine Science and Engineering, 2024, 12(2): 316.

[] Xu L, Teoh SS, Ibrahim H. A deep learning approach for electric motor fault diagnosis based on modified InceptionV3 Scientific Reports, 2024, 14(1): 12344.

[] Yeo R. Surveying the underside of an Artic ice ridge using a man-portable GAVIA AUV deployed through the ice OCEANS, IEEE, 2007, 2007: 1-8

[] Yoshida H, Hyakudome T, Ishibashi S. Yumeiruka-The AUV Equipped with an X-type Canard Rudder The Twenty-third International Offshore and Polar Engineering Conference, 2013 397-401

[] Yuan C, Shuai C, Ma J, Fang Y. An efficient control allocation algorithm for over-actuated AUVs trajectory tracking with fault-tolerant control Ocean Engineering, 2023, 273: 113976.

[] Žarković M, Stojković Z. Analysis of artificial intelligence expert systems for power transformer condition monitoring and diagnostics Electric Power Systems Research, 2017, 149: 125-136.

[] Zhai JQ, Yang X, Cheng YQ, Li L (2021) A review of the application of machine learning in the field of fault detection and diagnosis. Computer Measurement and Control (3): 1–9. DOI: https://doi.org/10.16526/j.cnki.11-4762/tp.2021.03.001

[] Zhang MJ, Yin BJ, Liu WX, Wang YJ. Feature extraction and fusion for thruster faults of AUV with random disturbance Journal of Huazhong University of Science and Technology (Nature Science Edition), 2015, 43(6): 22-26(54. (in Chinese))

[] Zhao H, Gao Y, Deng W. Defect Detection Using Shuffle Net-CA-SSD Lightweight Network for Turbine Blades in IoT IEEE Internet of Things Journal, 2024, 11(20): 32804-32812.