Stackelberg game-based anti-disturbance control for unmanned surface vessels via integrative reinforcement learning

Abstract

In the navigation of unmanned surface vessels (USVs), external disturbances, particularly ocean waves, frequently induce deviations from the desired trajectory. To mitigate these challenges, we propose a novel disturbance rejection control strategy based on Stackelberg game theory, designed to address unmodeled system dynamics, complex environmental conditions, and other external perturbations. This approach incorporates several key innovations. First, we introduce a velocity error dynamic system coupled with a non-cooperative Stackelberg game model, where the USV's control inputs (as the leader) and external disturbances (as the follower) interact within an alternating update framework. This leader-follower interaction facilitates the joint optimization of both the disturbance rejection and performance-optimal control strategies, enhancing the USV's tracking accuracy while maximizing its disturbance rejection capacity. Second, we rigorously verify the existence of a cooperative optimal solution through an analysis of the Nash equilibrium under sequential decision-making between the leader and follower. Building on this, integral reinforcement learning and neural networks are employed to approximate the optimal Stackelberg solution. The boundedness and convergence of the proposed approach are validated using Lyapunov functions, ensuring stability and optimal performance under dynamic operating conditions. Finally, simulation results confirm the efficacy of the proposed strategy, demonstrating its ability to concurrently optimize control robustness and performance - such as minimizing tracking error and energy consumption - when confronted with unmodeled dynamics and external disturbances.

Keywords

Unmanned surface vehicles / integral reinforcement learning / Stackelberg game / anti-disturbance control