A digital twins enabled underwater intelligent internet vehicle path planning system via reinforcement learning and edge computing

Abstract

The Autonomous Underwater Glider (AUG) is a kind of prevailing underwater intelligent internet vehicle and occupies a dominant position in industrial applications, in which path planning is an essential problem. Due to the complexity and variability of the ocean, accurate environment modeling and flexible path planning algorithms are pivotal challenges. The traditional models mainly utilize mathematical functions, which are not complete and reliable. Most existing path planning algorithms depend on the environment and lack flexibility. To overcome these challenges, we propose a path planning system for underwater intelligent internet vehicles. It applies digital twins and sensor data to map the real ocean environment to a virtual digital space, which provides a comprehensive and reliable environment for path simulation. We design a value-based reinforcement learning path planning algorithm and explore the optimal network structure parameters. The path simulation is controlled by a closed-loop model integrated into the terminal vehicle through edge computing. The integration of state input enriches the learning of neural networks and helps to improve generalization and flexibility. The task-related reward function promotes the rapid convergence of the training. The experimental results prove that our reinforcement learning based path planning algorithm has great flexibility and can effectively adapt to a variety of different ocean conditions.

Keywords

Digital twins / Reinforcement learning / Edge computing / Underwater intelligent internet vehicle / Path planning

References

[1] D. Deng, X. Li, V. Menon, M.J. Piran, H. Chen, M.A. Jan, Learning-based Joint UAV Trajectory and Power Allocation Optimization for Secure IoT Networks, Digital Communications and Networks 8 (4) (2022) 415-421.

[2] Q. Wang, H.-N. Dai, Q. Wang, M.K. Shukla, W. Zhang, C.G. Soares, On connectivity of UAV-assisted data acquisition for underwater internet of things, IEEE Internet Things J. 7 (6) (2020) 5371-5385.

[3] H. Lan, Y. Lv, J. Jin, J. Li, D. Sun, Z. Yang, Acoustical observation with multiple wave gliders for internet of underwater things, IEEE Internet Things J. 8 (4) (2021) 2814-2825.

[4] Y. Zhang, B. Kieft, B.W. Hobson, B.-Y. Raanan, S.S. Urmy, K.J. Pitz, C.M. Preston, B. Roman, K.J. Benoit-Bird, J.M. Birch, F.P. Chavez, C.A. Scholin, Persistent sampling of vertically migrating biological layers by an autonomous underwater vehicle within the beam of a seabed-mounted echosounder, IEEE J. Ocean. Eng. 46 (2) (2021) 497-508.

[5] L. Feng, Z. Lv, G. Guo, H. Song, Pheromone based alternative route planning, Digit. Commun. Netw. 2 (3) (2016) 151-158.

[6] O.A. Gbadamosi, D.R. Aremu, Design of a modified Dijkstra's algorithm for finding alternate routes for shortest-path problems with huge costs, in: 2020 International Conference in Mathematics, Computer Engineering and Computer Science (ICMCECS), 2020, pp. 1-6.

[7] T. Zheng, Y. Xu, D. Zheng,AGV path planning based on improved a-star algorithm, in: 2019 IEEE 3rd Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), 2019, pp. 1534-1538.

[8] H. Huang, P. Huang, S. Zhong, T. Long, S. Wang, E. Qiang, Y. Zhong, L. He,Dynamic path planning based on improved D* algorithms of gaode map, in: 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), 2019, pp. 1121-1124.

[9] C. Zhang, C. Hu, J. Feng, Z. Liu, Y. Zhou, Z. Zhang, A self-heuristic ant-based method for path planning of unmanned aerial vehicle in complex 3-D space with dense u-type obstacles, IEEE Access 7 (2019) 150775-150791.

[10] P. Wu, T. Li, G. Song,UCAV path planning based on improved chaotic particle swarm optimization, in:2020 Chinese Automation Congress (CAC), 2020, pp. 1069-1073.

[11] V. Roberge, M. Tarbouchi, G. Labonté, Fast genetic algorithm path planner for fixed-wing military UAV using GPU, IEEE Trans. Aero. Electron. Syst. 54 (5) (2018) 2105-2117.

[12] J. Yang, M. Xi, B. Jiang, J. Man, Q. Meng, B. Li, FADN: fully connected attitude detection network based on industrial video, IEEE Trans. Ind. Inf. 17 (3) (2020) 2011-2020.

[13] Y. Wang, J. Wang, W. Zhang, Y. Zhan, S. Guo, Q. Zheng, X. Wang, A survey on deploying mobile deep learning applications: a aystemic and technical perspective, Digit. Commun. Netw. 8 (1) (2022) 1-17.

[14] D. Jiang, Y. Wang, Z. Lv, S. Qi, S. Singh,Big data analysis based network behavior insight of cellular networks for industry 4.0 applications, IEEE Trans. Ind. Inf. 16 (2)(2019) 1310-1320.

[15] L. Qiao, S. Dang, B. Shihada, M.-S. Alouini, R. Nowak, Z. Lv, Can blockchain link the future? Digit. Commun. Netw. 8 (5) (2021) 2352-8648.

[16] X. Li, B. He, Y. Zhou, G. Li, Multisource model-driven digital twin system of robotic assembly, IEEE Syst. J. 15 (1) (2021) 114-123.

[17] C. Gehrmann, M. Gunnarsson, A digital twin based industrial automation and control system security architecture, IEEE Trans. Ind. Inf. 16 (1) (2020) 669-680.

[18] Z. Lv, B. Hu, H. Lv, Infrastructure monitoring and operation for smart cities based on IoT system, IEEE Trans. Ind. Inf. 16 (3) (2019) 1957-1962.

[19] W. Qin, S. Chen, M. Peng, Recent advances in industrial internet: insights and challenges, Digit. Commun. Netw. 6 (1) (2020) 1-13.

[20] X. Li, J. Wan, H.-N. Dai, M. Imran, M. Xia, A. Celesti, A hybrid computing solution and resource scheduling strategy for edge computing in smart manufacturing, IEEE Trans. Ind. Inf. 15 (7) (2019) 4225-4234.

[21] J. Chen, S. Chen, S. Luo, Q. Wang, B. Cao, X. Li, An intelligent task offloading algorithm (iTOA) for UAV edge computing network, Digit. Commun. Netw. 6 (4)(2020) 433-443.

[22] Z. Lv, W. Xiu, Interaction of edge-cloud computing based on SDN and NFV for next generation IoT, IEEE Internet Things J. 7 (7) (2019) 5706-5712.

[23] A. Ghadirzadeh, X. Chen, W. Yin, Z. Yi, M. Björkman, D. Kragic, Human-centered collaborative robots with deep reinforcement learning, IEEE Rob. Autom. Lett. 6 (2)(2021) 566-571.

[24] L. Huang, X. Feng, C. Zhang, L. Qian, Y. Wu, Deep reinforcement learning-based joint task offloading and bandwidth allocation for multi-user mobile edge computing, Digit. Commun. Netw. 5 (1) (2019) 10-17.

[25] R. Wang, X. Jiang, Y. Zhou, Z. Li, D. Wu, T. Tang, A. Fedotov, V. Badenko, c, Digital Communications and Networks 8 (1) (2022) 1-17.

[26] M. Volodymyr, K. Koray, S. David, A.A. Rusu, V. Joel, M.G. Bellemare, G. Alex, R. Martin, A.K. Fidjeland, O. Georg, Human-level control through deep reinforcement learning, Nature (518) (2015) 529-533.

[27] H. Van Hasselt, A. Guez, D. Silver,Deep reinforcement learning with double q-learning, in:Proceedings of the AAAI Conference on Artificial Intelligence, 2016, pp. 1-12.

[28] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, N. Freitas, Dueling network architectures for deep reinforcement learning, in: International Conference on Machine Learning, PMLR, 2016, pp. 1995-2003.

[29] F. Tao, H. Zhang, A. Liu, A.Y. Nee, Digital twin in industry: state-of-the-art, IEEE Trans. Ind. Inf. 15 (4) (2018) 2405-2415.

[30] F. Tao, F. Sui, A. Liu, Q. Qi, M. Zhang, B. Song, Z. Guo, S.C.-Y. Lu, A.Y. Nee, Digital twin-driven product design framework, Int. J. Prod. Res. 57 (12) (2019) 3935-3953.

[31] L.I. Hatledal, A. Styve, G. Hovland, H. Zhang, A language and platform independent co-simulation framework based on the functional mock-up interface, IEEE Access 7 (2019) 109328-109339.

[32] T. Mukherjee, T. DebRoy, A digital twin for rapid qualification of 3D printed metallic components, Appl. Mater. Today 14 (2019) 59-65.

[33] L. Kent, C. Snider, B. Hicks, Early stage digital-physical twinning to engage citizens with city planning and design, in: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), IEEE, 2019, pp. 1014-1015.

[34] A. Francisco, N. Mohammadi, J.E. Taylor, Smart city digital twin-enabled energy management: toward real-time urban building energy benchmarking, J. Manag. Eng. 36 (2) (2020) 04019045.

[35] N. Mohamed, J. Al-Jaroodi, S. Lazarova-Molnar,Leveraging the capabilities of industry 4.0 for improving energy efficiency in smart factories, IEEE Access 7 (2019) 18008-18020.

[36] M. De Donno, K. Tange, N. Dragoni, Foundations and evolution of modern computing paradigms: cloud, IoT, edge, and fog, IEEE Access 7 (2019) 150936-150948.

[37] Y. Li, H. Ma, L. Wang, S. Mao, G. Wang, Optimized content caching and user association for edge computing in densely deployed heterogeneous networks, IEEE Trans. Mobile Comput. 21 (6) (2022) 2130-2142.

[38] M. Goudarzi, H. Wu, M. Palaniswami, R. Buyya, An application placement technique for concurrent IoT applications in edge and fog computing environments, IEEE Trans. Mobile Comput. 20 (4) (2021) 1298-1311.

[39] S. Xia, Z. Yao, Y. Li, S. Mao, Online distributed offloading and computing resource management with energy harvesting for heterogeneous MEC-enabled IoT, IEEE Trans. Wireless Commun. 20 (10) (2021) 6743-6757.

[40] B. Li, Q. He, F. Chen, H. Jin, Y. Xiang, Y. Yang, Auditing cache data integrity in the edge computing environment, IEEE Trans. Parallel Distr. Syst. 32 (5) (2021) 1210-1223.

[41] Z. Chen, W. Liao, K. Hua, C. Lu, W. Yu, Towards asynchronous federated learning for heterogeneous edge-powered internet of things, Digital Communications and Networks 7 (3) (2021) 317-326.

[42] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al., A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, Science 362 (6419) (2018) 1140-1144.

[43] O. Vinyals, I. Babuschkin, W.M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature 575 (7782) (2019) 350-354.

[44] D. Ye, G. Chen, W. Zhang, S. Chen, B. Yuan, B. Liu, J. Chen, Z. Liu, F. Qiu, H. Yu, et al., Towards playing full moba games with deep reinforcement learning, Adv. Neural Inf. Process. Syst. 33 (2020) 621-632.

[45] Z. Ni, S. Paul, A multistage game in smart grid security: a reinforcement learning solution, IEEE Transact. Neural Networks Learn. Syst. 30 (9) (2019) 2684-2695.

[46] Y. Xiao, L. Xiao, X. Lu, H. Zhang, S. Yu, H.V. Poor, Deep-reinforcement-learning-based user profile perturbation for privacy-aware recommendation, IEEE Internet Things J. 8 (6) (2021) 4560-4568.

[47] Z. Zhang, Y.-S. Ong, D. Wang, B. Xue, A collaborative multiagent reinforcement learning method based on policy gradient potential, IEEE Trans. Cybern. 51 (2)(2021) 1015-1027.

[48] T. Chu, J. Wang, L. Codecà, Z. Li, Multi-agent deep reinforcement learning for large-scale traffic signal control, IEEE Trans. Intell. Transport. Syst. 21 (3) (2019) 1086-1095.