Path Planning for Unmanned Surface Vehicles in Dynamic Environments Based on Artificial Potential Field and Global Guided Reinforcement Learning

Abstract

For unmanned surface vehicles (USVs), how to find an effective, feasible path that substantially improves mission success rates and time efficiency in dynamic marine environments is a critical issue. To address the path planning problem for USVs using deep reinforcement learning (DRL) in dynamic ocean environments, an improved algorithm based on Deep Q-Networks (DQN) is proposed, which is called Fast Guided Deep Q-Network Algorithm (FG-DQN). This algorithm combines DQN with the artificial potential field (APF) method and uses the A* algorithm to initialize a guiding path in a global static environment and to provide prior knowledge for the USVs. Additionally, the configuration of the reward function using APF and the guiding path effectively reduces the frequency of random movements during the early exploration phase of the DQN algorithm, which accelerates convergence, improves the computational efficiency of path planning, and increases path safety. Finally, the performance of the presented algorithm is validated through experiments in a 2D environment. Compared with traditional reinforcement learning methods such as Q-learning and Sarsa, as well as the original DQN algorithm, FG-DQN is more effective for USV path planning.

Keywords

Deep reinforcement learning / Path planning / Unmanned surface vehicles / Fast guided deep Q-Network algorithm

References

[1] ChengC, ShaQ, HeB, LiG. Path planning and obstacle avoidance for AUV: A review. Ocean Engineering, 2021, 235: 109355

[2] CaiK, WangC, ChengJ, De SilvaCW, MengMQH. Mobile robot path planning in dynamic environments: A survey. Instrumentation, 2020, 6(2): 90-100

[3] CobbHG, GrefenstetteJJ. Genetic algorithms for tracking changing environments. Proceedings of the 5th International Conference on Genetic Algorithms, 1993523-530

[4] ColorniA, DorigoM, ManiezzoV. Distributed optimization by ant colonies. Proceedings of the First European Conference on Artificial Life, 1991134-142Vol. 142

[5] DechterR, PearlJ. Generalized best-first search strategies and the optimality of A*. Journal of the ACM (JACM), 1985, 32(3): 505-536

[6] DingMH, LiuH, ZhengGH. Determining a stationary mean field game system from full/partial boundary measurement. SIAM Journal on Mathematical Analysis, 2025, 57(1): 661-681

[7] EberhartR, KennedyJ. A new optimizer using particle swarm theory. MHS’95, Proceedings of the Sixth International Symposium on Micro Machine and Human Science, 199539-43

[8] GaugueA, MenardM, MigotE, BourcierP, GaschetC. Development of an aquatic USV with high communication capability for environmental surveillance. OCEANS 2019-Marseille, 20191-8

[9] HongSM, NamKS, RyuJD, LeeDG, HaKN. Development and field test of unmanned marine vehicle (USV/UUV) with cable. IEEE Access, 2020, 8: 193347-193355

[10] HuangZ, LinH, ZhangG. The USV path planning based on an improved DQN algorithm. 2021 International Conference on Networking, Communications and Information Technology (NetCIT), 2021162-166

[11] HaoB, DuH, YanZ. A path planning approach for unmanned surface vehicles based on dynamic and fast Q-learning. Ocean Engineering, 2023, 270: 113632

[12] HuX, WuH, SunQ, LiuJ. Robot time optimal trajectory planning based on improved simplified particle swarm optimization algorithm. IEEE Access, 2023, 11: 44496-44508

[13] HeK, FeiY, TengX, ChuX, MaZ. Optimal path planning for underwater robots based on improved ant colony algorithm. 2022 IEEE International Conference on Unmanned Systems (ICUS), 20221118-1123

[14] ImanuvilovO, HongyuL, YamamotoM. Unique continuation for a mean field game system. Applied Mathematics Letters, 2023, 145: 108757

[15] JiangL, HuangH, DingZ. Path planning for intelligent robots based on deep Q-learning with experience replay and heuristic knowledge. IEEE/CAA Journal of Automatica Sinica, 2019, 7(4): 1179-1189

[16] KhatibO. Real-time obstacle avoidance for manipulators and mobile robots. The International Journal of Robotics Research, 1986, 5(1): 90-98

[17] LiY, ZhaoJ, ChenZ, XiongG, LiuS. A robot path planning method based on improved genetic algorithm and improved dynamic window approach. Sustainability, 2023, 15(5): 4656

[18] LiuH, LoCW. Determining state space anomalies in mean field games. Nonlinearity, 2025, 38(2): 025010

[19] LiuH, MouC, ZhangS. Inverse problems for mean field games. Inverse Problems, 2023, 39(8): 085003

[20] Liu H, Zhang S (2025) Simultaneously recovering running cost and Hamiltonian in mean field games system. arXiv preprint arXiv: 2303.13096. https://doi.org/10.48550/arXiv.2303.13096

[21] LiuL, ShanQ, XuQ. USVs path planning for maritime search and rescue based on POS-DQN: Probability of success-deep Q-Network. Journal of Marine Science and Engineering, 2024, 12(7): 1158

[22] Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing Atari with deep reinforcement learning. arXiv preprint arXiv: 1312.5602. https://doi.org/10.48550/arXiv.1312.5602

[23] MinskyMLTheory of neural-analog reinforcement systems and its application to the brain-model problem, 1954, Princeton, USA, Princeton University

[24] MacTT, CopotC, TranDT, De KeyserR. Heuristic approaches in robot path planning: A survey. Robotics and Autonomous Systems, 2016, 86: 13-28

[25] SangH, YouY, SunX, ZhouY, LiuF. The hybrid path planning algorithm based on improved A* and artificial potential field for unmanned surface vehicle formations. Ocean Engineering, 2021, 223: 108709

[26] TamangJ, NkapkopJDD, IjazMF, PrasadPK, TsafackN, SahaA, KengneJ, SonY. Dynamical properties of ion-acoustic waves in space plasma and its application to image encryption. IEEE Access, 2021, 9: 18762-18782

[27] WatkinsCJ, DayanP. Q-learning. Machine Learning, 1992, 8: 279-292

[28] YangJ, HuoJ, XiM, HeJ, LiZ, SongHH. A time-saving path planning scheme for autonomous underwater vehicles with complex underwater conditions. IEEE Internet of Things Journal, 2022, 10(2): 1001-1013

[29] YuK, LiangXF, LiMZ, ChenZ, YaoYL, LiX, ZhaoZX, TengY. USV path planning method with velocity variation and global optimisation based on AIS service platform. Ocean Engineering, 2021, 236: 109560

[30] ZhangH, HuangY, QinH, GengZ. USV search mission planning methodology for lost target rescue on sea. Electronics, 2023, 12(22): 4584

[31] ZhaiH, WangW, ZhangW, LiQ. Path planning algorithms for USVs via deep reinforcement learning. 2021 China Automation Congress (CAC), 20214281-4286

[32] ZhangM, CaiW, PangL. Predator-prey reward based Q-learning coverage path planning for mobile robot. IEEE Access, 2023, 11: 29673-29683

[33] ZengZ, SammutK, LianL, HeF, LammasA, TangY. A comparison of optimization techniques for AUV path planning in environments with ocean currents. Robotics and Autonomous Systems, 2016, 82: 61-72

RIGHTS & PERMISSIONS

Harbin Engineering University and Springer-Verlag GmbH Germany, part of Springer Nature