R3LOOP: Residual Recurrent Reinforcement Learning for Cross-Opponent-Strategy Orbital Pursuit Decision

Wei Dong , Jiajie Wang , Chen Chen , Fang Deng , Zhengtao Ding

ENGINEERING Artificial Intelligence ››

PDF (27669KB)
ENGINEERING Artificial Intelligence ›› DOI: 10.2738/ENGAI.2026.0004
Original Research
R3LOOP: Residual Recurrent Reinforcement Learning for Cross-Opponent-Strategy Orbital Pursuit Decision
Author information +
History +
PDF (27669KB)

Abstract

This paper proposes a residual recurrent reinforcement learning framework, i.e., R3LOOP, for orbital pursuit decision-making with cross-opponent-strategy generalization. First, a linear quadratic regulator (LQR)-based nominal pursuit prior is constructed from the Clohessy–Wiltshire (CW) relative motion model to provide a baseline impulsive action for the pursuer. Then, a proximal policy optimization (PPO) policy is introduced to learn a residual correction to refine the LQR action, which reduces the search difficulty of pure reinforcement learning (RL) and improves the adaptability of the pursuer in adversarial interactions. Finally, to address the challenge of cross-opponent generalization, a gated recurrent unit (GRU)-based residual policy and an opponent-pool training mechanism are developed. Unlike fixed-opponent learning mechanisms, the proposed method aims to train a single pursuer policy against multiple evader strategies. This hybrid design combines the model-based pursuit prior with learning-based residual correction, enabling the learning policy to refine the nominal pursuit action without learning the policy from scratch. Numerical simulations are provided to show the effectiveness and advantages of the proposed framework.

Graphical abstract

Keywords

Orbital game / Recurrent reinforcement learning / Opponent pool / Cross-opponent-strategy generalization

Highlight

● A cross-opponent orbital pursuit decision problem is studied by training a single pursuer policy against multiple evader strategies.

● An LQR-guided residual reinforcement learning structure learns corrections to nominal pursuit actions, reducing the search difficulty of pure reinforcement learning.

● A recurrent policy with multi-evader training uses historical interaction data to improve adaptability across different evader strategies.

Cite this article

Download citation ▾
Wei Dong, Jiajie Wang, Chen Chen, Fang Deng, Zhengtao Ding. R3LOOP: Residual Recurrent Reinforcement Learning for Cross-Opponent-Strategy Orbital Pursuit Decision. ENGINEERING Artificial Intelligence DOI:10.2738/ENGAI.2026.0004

登录浏览全文

4963

注册一个新账户 忘记密码

1 Introduction

Autonomous decision-making for non-cooperative spacecraft has become an important requirement for future space operations, such as on-orbit service, space situational awareness, active debris removal, and spacecraft safety management. With the increasing complexity of space missions, the interaction between the pursuer and the evader becomes more dynamic, adversarial, and uncertain. The orbital pursuit-evasion game is essentially a complex sequential decision-making problem. Designing a practical decision-making method with good adaptability and generalization is significant but challenging in this field [1–5].

For the orbital pursuit-evasion game, early studies mainly focused on classical game-theoretic and control methods [6–9]. The authors in Ref. [10] modeled the problem as a three-dimensional orbital pursuit-evasion game and systematically studied its numerical solution, which laid a foundation for later modeling and solution methods. In Ref. [11], the saddle-point solution of the orbital pursuit-evasion game under J2-perturbed dynamics was investigated, and the classical game-theoretic framework was extended to perturbed orbital scenarios. The pursuit-evasion problem under incomplete information was studied in Ref. [12] and a switching-strategy method was proposed to improve the applicability in non-ideal information conditions. A parameter-optimized pursuit strategy was developed for orbital games in Ref. [13] where an effective pursuit decision framework was provided when the opponent information was unavailable or imperfect. In Ref. [14], J2 perturbations and control constraints were introduced into the orbital pursuit-evasion formulation, making the model closer to practical engineering scenarios. On this basis, pursuit-evasion methods have gradually developed toward predictive control and hierarchical game frameworks. The authors in Ref. [15] combined predictive pursuit-evasion control with the model predictive control framework for approaching non-cooperative space targets. In Ref. [16], a model predictive Stackelberg solution was proposed for the orbital pursuit-evasion game to improve real-time decision-making capability. In Ref. [17], the impulsive orbital game was formulated in a Stackelberg equilibrium framework, which promoted the development of pursuit-evasion methods under hierarchical game structures. Notably, existing classical methods mostly rely on explicit and simplified orbital dynamics. The feasibility and adaptability under practical complex dynamics and unknown hostile behavior are challenging to ensure.

With the rapid development of deep learning and RL, intelligent decision-making methods have received increasing attention for orbital pursuit-evasion games [18–22]. In Ref. [23], deep RL was used to construct a near-optimal interception strategy, and the feasibility of learning-based closed-loop decision-making was verified in orbital pursuit-evasion scenarios. A deep-RL-based pursuit strategy was proposed in Ref. [24] for orbital interception against random evasion. The authors in Ref. [25] introduced a Q-learning method from the perspective of a nonzero-sum game to solve orbital pursuit-evasion control problems. A learning-based algorithm named PRD-MADDPG was developed in Ref. [26] for the orbital pursuit-evasion game with impulsive maneuvers. The authors in Ref. [27] extended RL-based decision-making to the orbital pursuit-evasion game in elliptical orbits. In Ref. [28], the multi-agent orbital pursuit-evasion problem under sparse rewards was investigated, and a hierarchical RL framework based on hindsight experience replay was proposed to improve training convergence and maneuver strategy learning. More recently, the two-player orbital pursuit-evasion problem under impulsive thrust control was studied in Ref. [29] and an impulse-like constraint was introduced to incorporate impulsive maneuvers into a differential-game framework. In Ref. [30], a continuous-time hierarchical RL framework was proposed for pursuit decision and its potential in long-horizon pursuit tasks was demonstrated. Although RL can learn a policy through interaction without an explicit model, training remains computationally expensive because of the large policy space and inefficient early exploration.

In recent years, researchers have increasingly combined model-based methods with RL to improve training efficiency [31]. Although this approach has received less attention in orbital pursuit-evasion games, it has shown promise in related fields such as robot control. For complex robot control tasks, the authors in Ref. [32] proposed residual policy learning, which learns a residual policy on top of an existing controller. This method improves learning efficiency and task performance by combining baseline control with learned correction. Similarly, in Ref. [33], conventional feedback control was directly combined with deep RL. The method used residual RL to compensate for contact effects, friction, and modeling errors and showed good performance in real robot control. Furthermore, the authors in Ref. [34] proposed a hybrid framework that first imitated the model predictive control and then used RL for fine-tuning. This work showed that a model-based controller can serve not only as a prior controller at deployment but also as an expert signal during training. In a recent work [35], a model-based whole-body controller was further integrated into an RL feedback policy to improve both accuracy and robustness. A similar hybrid design was developed in [36] to address spacecraft proximity operations. Note that the above results are based on a specific scenario or a cooperative target. Existing hybrid methods generally refine a nominal controller for a specific task or fixed interaction setting. For the orbital pursuit-evasion game in adversarial scenarios, the design of an efficient and generalized intelligent decision-making method against an uncooperative target remains an open challenge.

Inspired by the above observations, this paper proposes R3LOOP, an LQR-guided residual recurrent reinforcement learning framework for three-dimensional orbital pursuit-evasion. The main contributions of this paper are threefold:

(1) A cross-opponent orbital pursuit decision problem is studied. Different from fixed-opponent settings, this work trains a single pursuer policy against multiple evader strategies and evaluates its performance across different opponents.

(2) A prior-guided residual reinforcement learning structure is proposed. The LQR controller provides a model-based nominal pursuit action. The PPO policy learns a residual correction to refine this action. This design reduces the difficulty of pure RL search and improves the adaptability of the pursuer in adversarial interactions.

(3) A recurrent policy with multi-evader training is developed. The recurrent module uses historical interaction data to represent evader behavior. Multiple evader strategies are used during training. This improves the adaptability of one pursuer policy across different evader strategies.

The remainder of this paper is organized as follows. Section 2 presents the problem formulation and mathematical model. Section 3 introduces the proposed decision-making framework for the orbital pursuit-evasion game. Section 4 provides numerical simulations and performance analysis. Finally, Section 5 concludes this paper.

2 Problem Formulation

This section introduces the mission scenario, dynamic model and constraints of the three-dimensional orbital pursuit-evasion problem.

2.1 Mission scenario and orbital dynamics

In this paper, a pursuit-evasion problem in the vicinity of the geostationary Earth orbit (GEO) is considered. Both the pursuer and the evader perform orbital control through impulsive maneuvers. To describe the pursuit-evasion process, a local-vertical local-horizontal (LVLH) coordinate frame is established. A notional reference spacecraft is assumed to move on a circular GEO, and the LVLH coordinate frame is attached to this reference spacecraft, whose center is taken as the origin. The x-axis is the radial direction and points from the Earth’s center to the spacecraft, the y-axis is aligned with the velocity direction, and the z-axis completes a right-handed orthogonal coordinate system. In this frame, the states of the pursuer and evader are defined relative to the notional reference satellite. The configuration of the LVLH coordinate frame and the definition of the relative position vector are shown in Fig. 1.

Similar to common practice [26,37], the relative distance between the pursuer and the evader is assumed to remain much smaller than the GEO orbital radius throughout the engagement. Therefore, the Clohessy–Wiltshire (CW) linearized model is adopted. For spacecraft i∈{p,e}, where p and e denote the pursuer and the evader, the relative motion equations are

x¨i−2ny˙i−3n2xi=ux,i,y¨i+2nx˙i=uy,i,z¨i+n2zi=uz,i,

where xi, yi, and zi are the position components of spacecraft i along the three axes of the LVLH frame, ux,i, uy,i, and uz,i are the corresponding continuous acceleration components, and n denotes the mean orbital angular velocity of the target spacecraft calculated by

n=μ/R3,

where μ is the Earth’s gravitational parameter and R is the GEO radius of the target spacecraft. For GEO, R≈4.2164×107m, yielding n≈7.2921×10−5rad/s.

Define the state vector as

Xi(t)=[ri(t)vi(t)]=[xiyizix˙iy˙iz˙i]T,

where Xi(t), ri(t), and vi(t) denote the state, position, and velocity of spacecraft i, respectively. Then, the CW dynamics can be written in the standard state-space form as

X˙i(t)=AXi(t)+Bui(t),

where

A=[03×3I3×3A21A22],B=[03×3I3×3],

with

A21=[3n20000000−n2],A22=[02n0−2n00000],

and ui(t)=[ux,i(t),uy,i(t),uz,i(t)]T is the continuous control acceleration vector of spacecraft i.

Given the initial state Xi(t0), the future state at any time t can be calculated by

Xi(t)=Φ(t,t0)Xi(t0)+∫t0tΦ(t,τ)Bui(τ)dτ

where Φ(t,t0) is the state transition matrix given by

Φ(t,t0)=[Φrr(t,t0)Φrv(t,t0)Φvr(t,t0)Φvv(t,t0)],

For the CW dynamics, the four submatrices of Φ(t,t0) are given by

Φrr(t,t0)=[4−3cos⁡n(t−t0)006(sin⁡n(t−t0)−n(t−t0))1000cos⁡n(t−t0)],Φrv(t,t0)=[sin⁡n(t−t0)n2(1−cos⁡n(t−t0))n02(cos⁡n(t−t0)−1)n4sin⁡n(t−t0)−3n(t−t0)n000sin⁡n(t−t0)n],Φvr(t,t0)=[3nsin⁡n(t−t0)006n(cos⁡n(t−t0)−1)0000−nsin⁡n(t−t0)],Φvv(t,t0)=[cos⁡n(t−t0)2sin⁡n(t−t0)0−2sin⁡n(t−t0)4cos⁡n(t−t0)−3000cos⁡n(t−t0)].

In this study, both pursuer and evader spacecraft use impulsive control to maneuver. When an impulsive velocity increment Δvi is applied at time t0, the post-impulse state is

Xi(t0+)=[ri(t0)vi(t0)+Δvi].

Then, the state at any later time t is

Xi(t)=Φ(t,t0)Xi(t0+)=Φ(t,t0)Xi(t0)+Φv(t,t0)Δvi,

and the terminal state at the terminal time tf is

Xi(tf)=Φ(tf,t0)Xi(t0)+Φv(tf,t0)Δvi,

where Δvi denotes the impulsive velocity increment applied to spacecraft i at time t0.

2.2 Constraints and termination conditions

Based on the above relative motion model, the orbital pursuit-evasion process can be formulated as a discrete-time sequential game using a fixed time step Δt. At the beginning of each step, the pursuer and the evader apply impulsive thrusts. Each episode contains at most N time steps, and the maximum pursuit time is

tmax=NΔt.

The k-th decision instant is

t(k)=kΔt.

After each impulse, the relative position and velocity of both agents propagate according to Eq. (11). The overall process of the pursuit-evasion game is shown in Fig. 2.

In the orbital pursuit-evasion game, both spacecraft are subject to the following constraints.

(1) Single-step per-axis impulsive magnitude constraint: at each time step k, the impulsive velocity increment of spacecraft i is denoted by Δvi(k). Define the per-axis impulse-bound vector as

ηi=[ηx,iηy,iηz,i]T,

where ηq,i is the impulse bound along axis q∈{x,y,z} for spacecraft i, i∈{p,e}. The componentwise constraint is

|Δvq,i(k)|⩽ηq,i,q∈{x,y,z},i∈{p,e}.

(2) Cumulative velocity-increment constraint: the per-step maneuver cost and cumulative maneuver cost are defined as

ci(k)=‖Δvi(k)‖1=|Δvx,i(k)|+|Δvy,i(k)|+|Δvz,i(k)|.Ci(k)=∑ℓ=0k−1ci(ℓ).

Let C¯i denote the maximum allowable cumulative maneuver cost. The constraint is

Ci(k)⩽C¯i.

The cumulative maneuver cost Ci(k) is reported in m/s and is used as a maneuver-cost proxy rather than as a direct measure of propellant mass.

(3) Time constraint: the pursuit-evasion game is bounded by a finite time horizon as

tf=kfΔt⩽NΔt=tmax.

Here, kf denotes the number of decision intervals completed when the episode terminates, with 1≤kf≤N.

In addition, the orbital pursuit-evasion game terminates if one of the following two conditions holds:

(1) Unsuccessful termination: the mission fails when the pursuer cannot capture the evader within the given constraints. This case includes:

(a) the maximum pursuit time is reached, that is

{tf=tmaxd(kf)>dcap.

(b) the pursuer violates its cumulative maneuver-cost budget, that is

Cp(kf)⩾C¯p.

(2) Successful termination: the mission succeeds when the pursuer captures the evader before the maximum time and without violating the cumulative maneuver-cost budget, that is

{d(kf)⩽dcaptf⩽tmaxCp(kf)≤C¯p.

where d(k) denotes the relative distance between the pursuer and the evader, and dcap denotes the preset capture threshold.

3 R3LOOP for Orbital Pursuit Decision-Making

This section presents the proposed R3LOOP framework for the three-dimensional orbital pursuit-evasion game. The flowchart of the R3LOOP framework is shown in Fig. 3.

3.1 LQR-based baseline strategy

Note that a game-theoretic controller would require an explicit evader model or strategy assumption, which conflicts with the objective of adapting one pursuer to multiple evasion policies. Therefore, the LQR technique is used to derive an opponent-independent model-based prior, while opponent-dependent adaptation will be addressed by the residual policy in the next subsection.

For the nominal strategy design, the evader is assumed not to maneuver, i.e., Δve(k)=0. Under this ideal condition, by Eq. (11), the discrete-time relative dynamics can be written as

Xpe(k+1)=AdXpe(k)+BdΔvp(k),

where Xpe(k)=Xp(k)−Xe(k) is the relative state at t(k), and Δvp(k) is the pursuer impulse. Note that the two terms on the right-hand side represent the free propagation of the relative state and the state increment induced by the pursuer impulse, respectively. The matrices Ad and Bd are constructed from the state-transition submatrices in Eq. (9) as

Ad=[Φrr(t(k+1),t(k))Φrv(t(k+1),t(k))Φvr(t(k+1),t(k))Φvv(t(k+1),t(k))]Bd=[Φrv(t(k+1),t(k))Φvv(t(k+1),t(k))].

To obtain the baseline strategy, the quadratic performance index is defined as

J=∑k=0∞((Xpe(k))TQXpe(k)+(Δvp(k))TRΔvp(k)),

where Q and R are the state and control weighting matrices, respectively. The design objective is to minimize J subject to the discrete dynamics in Eq. (23).

By the LQR technique, the corresponding optimal value function is

V(Xpe)=XpeTPXpe,

where P satisfies the following discrete algebraic Riccati equation

P=Q+AdTPAd−AdTPBd(R+BdTPBd)−1BdTPAd.

The resulting state-feedback law is

ΔvpLQR,(k)=−KXpe(k).

where the feedback gain matrix is given by

K=(R+BdTPBd)−1BdTPAd.

The LQR baseline gives a model-based pursuit direction and reduces the search difficulty of the learning policy. In the actual implementation, the LQR command is further constrained by the single-step impulsive magnitude limit. Therefore, the LQR baseline should be understood as a nominal pursuit prior rather than a complete solution to the pursuit-evasion game.

3.2 Residual RL-based adaptive strategy

The nominal LQR designed in Subsection 3.1 is derived by assuming the evader remains non-maneuvering. However, during actual pursuit-evasion, the evader may actively perform unknown maneuvers. To handle such maneuvers, a PPO+LQR framework is introduced to correct the nominal LQR decision.

The environment state is explicitly defined as

s(k)=[rpe(k)vpe(k)rp(k)vp(k)re(k)ve(k)]∈R18.

The LQR impulse is further appended to form the policy input as

s′(k)=[(s(k))T,(ΔvpLQR,(k))T]T∈R21.

Before entering the network, the three position blocks, three velocity blocks, and LQR-impulse block are divided by fixed position, velocity, and impulse scales, respectively. Note that these scales should be appropriately chosen according to specific scenarios. Table 2 in Section 4 provides an exemplified case for the considered simulation settings.

The PPO algorithm adopts an actor-critic architecture and both the actor and the critic use standard feedforward neural networks. Based on the augmented input s′(k), the actor outputs the mean of the residual action distribution, and the critic estimates the corresponding state value.

In the action representation, the policy outputs a residual correction. For the Gaussian policy, the actor outputs the action mean μ(k), and the policy distribution is defined as

πθ(Δvpres,(k)∣s′(k))=N(μ(k),diag(σ2)).

The final control input applied to the pursuer is given by

Δvp(k)=ΔvpLQR,(k)+Δvpres,(k).

This structure keeps the LQR strategy and the PPO policy in the same impulsive control domain. The LQR strategy supplies the nominal pursuit action, while PPO learns an opponent-dependent residual correction.

During training, the policy is updated with the clipped PPO objective. Let the probability ratio between the new policy and the old policy be

ρ(k)(θ)=πθ(Δvpres,(k)∣s′(k))πθold(Δvpres,(k)∣s′(k)).

Then, the actor objective is

LCLIP(θ)=Ek[min(ρ(k)(θ)A^(k),clip(ρ(k)(θ),1−ϵ,1+ϵ)A^(k))].

The advantage function is computed by generalized advantage estimation (GAE) as

A^(k)=∑l=0N−k−1(γλ)lδ(k+l),

where A^(k) is the GAE advantage estimate at step k, and the temporal-difference error δ(k) is

δ(k)=r(k)+γVϕ(s′(k+1))−Vϕ(s′(k)).

The critic is trained by minimizing the state value estimation error

LV(ϕ)=(Vϕ(s′(k))−V^(k))2,

where V^(k) denotes the value target at step k.

Based on the above design, PPO refines the LQR baseline action through residual correction, thereby improving the adaptability of the pursuer in dynamic adversarial interactions.

3.3 Residual recurrent RL-based cross-opponent strategy

Although the residual RL framework proposed in Subsection 3.2 can improve the baseline action, it mainly uses current-step information if a feedforward policy is adopted. This setting is not sufficient for cross-opponent tasks. Different opponents may show different maneuver patterns over time. Therefore, a residual recurrent policy is introduced to extract historical interaction information and improve generalization against multiple opponents.

The actor adopts a GRU-based network for temporal modeling. The same augmented state s′(k) defined in Eq. (31) is used together with the GRU hidden state h(k), which contains the historical interaction information available at decision step k. As such, the recurrent policy generates the residual correction using both the current orbital information and the accumulated interaction history. To this end, the input feature is first obtained by

f(k)=σ(W1s′(k)+b1).

Then, the hidden state h(k+1) is updated by the GRU according to

ξ(k)=σ(Wξf(k)+Uξh(k)+bξ),

z(k)=σ(Wzf(k)+Uzh(k)+bz),

h~(k+1)=tanh⁡(Whf(k)+Uh(ξ(k)⊙h(k))+bh),

h(k+1)=(1−z(k))⊙h(k)+z(k)⊙h~(k+1),

where ξ(k) and z(k) denote the reset gate and update gate, respectively, h~(k+1) denotes the candidate hidden state, and ⊙ denotes the element-wise product.

Based on the updated hidden representation h(k+1), an additional feature transformation is introduced as

g(k)=σ(W2h(k+1)+b2).

The actor outputs the mean of the residual action distribution as

μ(k)=Wμg(k)+bμ.

Compared with the feedforward policy, the recurrent actor can use historical interaction information through the hidden state. This design helps the pursuer adjust its residual correction according to different opponent behaviors.

The critic estimates the state value based on the augmented input, which can be written as

Vϕ(s′(k))=fϕ(s′(k)),

where fϕ(⋅) denotes the value network. This value network is used to compute the PPO value loss and the advantage estimate.

Under the recurrent setting, the policy distribution is expressed as

πθ(Δvpres,(k)∣s′(k),h(k))=N(μ(k),diag(σ2)).

Accordingly, the probability ratio in PPO becomes

ρ(k)(θ)=πθ(Δvpres,(k)∣s′(k),h(k))πθold(Δvpres,(k)∣s′(k),h(k)).

Remark 1: at the beginning of each episode, the GRU hidden state should be initialized before the first decision step and will be subsequently propagated throughout the episode. During PPO updates, transitions from the same episode are preserved in temporal order and divided into contiguous sequences with a maximum length of L. For each sequence, the hidden state stored at its first transition is used as the initial hidden state.

To improve cross-opponent adaptability, an opponent-pool training mechanism is further introduced. The opponent pool is defined as

O={o1,o2,…,oM}.

In this work, the pool contains M different types of evaders, including learning-based evaders, model-based evaders, and non-maneuvering evaders. During each training episode, one opponent is sampled from the pool. This provides different state-transition distributions and prevents the pursuer from fitting only one fixed evader strategy.

Let the opponent sampling distribution associated with training episode e be

pe(o),o∈O.

The opponent for each episode is sampled according to the stage-dependent distribution pe(o). The sampling distribution varies across different training stages. The training objective for episode e can be written as

maxθEo∼pe(o)[G(θ;o)].

where G(θ;o) denotes the expected return of policy θ against opponent o.

Through the above residual recurrent policy and opponent-pool training, the proposed R3LOOP framework combines the model-based LQR prior with learning-based residual adaptation. The LQR baseline provides the main pursuit direction, the PPO residual refines this action, and the recurrent hidden state encodes historical interaction information. As a result, one pursuer policy can better adapt to different evader strategies.

3.4 Training reward function design

In addition to the strategy structure, the reward function is also a key factor in the proposed framework. According to the objectives and constraints of the orbital pursuit-evasion task, the reward function r(k) is constructed by four parts: the distance variation term rd(k), the time penalty term rτ(k), the terminal outcome term rf(k), and the residual-impulse penalty term rΔv(k). Therefore, the reward function can be written as

r(k)=rd(k)+rτ(k)+rf(k)+rΔv(k).

The distance variation term rd(k) is introduced to encourage the pursuer to continuously reduce the relative distance to the evader. Let d(k) denote the relative distance between the pursuer and the evader at decision step k. Then, this reward term is given by

rd(k)=α(d(k)−d(k+1)),

where α>0 is the distance reward coefficient. When the relative distance decreases within the current time step, this term is positive. Otherwise, it becomes negative. In this way, the policy receives immediate feedback that is consistent with the objective of approaching the target.

The time penalty term rτ(k) is introduced to prevent ineffective delay during the pursuit-evasion process and to encourage the pursuer to complete the capture task as quickly as possible under the given constraints. This term is defined as

rτ(k)=−κτ,

where κτ>0 is the constant time-penalty coefficient. Since this term is negative at every time step, the accumulated penalty increases as the episode becomes longer. Therefore, this term drives the policy to accomplish the task in fewer steps.

The terminal outcome term rf(k) is used to emphasize the final mission result and the importance of successful capture. A large positive reward is assigned when the pursuer satisfies the capture condition. A penalty is assigned when the episode ends in failure. This term is written as

rf(k)={rsucc,d(k+1)⩽dcap−rfail,mission failure0,otherwise,

where rsucc>0 denotes the capture reward, rfail>0 denotes the failure penalty, and dcap is the preset capture threshold. This term strengthens the final task objective, so that the policy is optimized not only for local gains during the process, but also for successful capture from a global perspective.

The residual-impulse penalty term rΔv(k) is used to suppress excessive control input. It can also constrain excessive corrections to the baseline strategy and avoid unnecessary additional maneuvers. Define the residual maneuver cost as

χpres,(k)=‖Δvpres,(k)‖1.

The residual-impulse penalty is

rΔv(k)=−βχpres,(k),

where β>0 is the residual-impulse penalty coefficient and ‖⋅‖1 denotes the sum of the absolute values of the three components.

By combining the above terms together, the final reward at each step k is written as

r(k)=α(d(k)−d(k+1))−κτ+rf(k)−βχpres,(k).

This reward design jointly considers pursuit efficiency, mission duration, terminal outcome, and control cost. It provides a training signal that is consistent with the objective of the orbital pursuit-evasion task.

The overall training procedure of the proposed framework is summarized in Algorithm 1.

4 Simulation Results

This section presents numerical simulations of the proposed framework. The results are designed to validate the residual correction mechanism and the cross-opponent generalization capability under the recurrent policy.

4.1 Simulation settings

Two groups of simulations are conducted under the scenario described in Section 2. The simulations adopt the same scenario setting, dynamic model, and evaluation criteria. At the beginning of each episode, the initial position of the evader is fixed at the origin, while the initial position of the pursuer is randomly generated within a spherical shell centered at the origin with a radius of 30–35 km. The pursuer and the evader apply impulsive thrusts simultaneously at each control instant, and each impulse is treated as an instantaneous velocity increment directly added to the spacecraft velocity. After the impulse is applied, the system evolves freely within one discrete decision period according to the relative motion dynamics. The decision period Δt=600s is the interval between consecutive action updates. Within each decision period, the orbital state is propagated every Δtp=30s, and the capture condition is checked at each propagation step; these steps do not introduce additional control decisions. In addition, both spacecraft are assumed to have full-state observability, that is, the relative position and relative velocity at the current control step are available at every decision instant.

During both training and testing, the reward function follows the unified form given in Section 3 and remains unchanged in all experiments. Unless otherwise specified, all subsequent experiments use the same parameter settings. The opponent pool contains LQR, PPO, soft actor-critic (SAC), and stationary evaders. The LQR evader applies a discrete-CW feedback impulse, while the stationary evader applies zero impulse. The independently trained PPO and SAC evaders receive the 18-dimensional state in Eq. (30) and output three-dimensional impulses; their parameters are fixed during pursuer training, and deterministic actions are used for evaluation. Each component of a maneuvering evader’s impulse is constrained to [−1, 1] m/s. The detailed simulation and training parameters are listed in Tables 1 and 2.

The curriculum uses three training intervals: 0−20%, 20%−66%, and 66%−100%. For the ordered opponent set (PPO, SAC, LQR, stationary), the corresponding sampling probabilities are (0.35,0.35,0.20,0.10), (0.20,0.25,0.50,0.05), and (0.20,0.35,0.40,0.05), respectively.

4.2 Residual RL effectiveness validation

This subsection isolates the effect of residual learning under single-opponent training. For each evader strategy, an independent PPO+LQR pursuer is trained against that evader and compared with the LQR and PPO baselines under identical simulation settings.

The simulations use the same settings as above. LQR is taken as the prior strategy. PPO is introduced to provide residual correction. Four representative evader policies are considered, including LQR, PPO, SAC, and a stationary policy. For each evader policy, a PPO+LQR pursuer model is trained separately. The trained model is then loaded for 500 Monte Carlo interception tests. The results are reported in Tables 3 and 4 and Figs. 4, 5 and 6.

Table 3 shows that the independent PPO policy can hardly complete the task under the considered setting. This is mainly because the terminal capture distance is small, while the CW update interval is relatively long. Although an intermediate capture check is performed every 30s during propagation, the independent PPO policy with a traditional reward design still fails in most cases. In contrast, the independent LQR policy shows different performance against different evader policies. It achieves 100% capture against the stationary evader, but only 2.8% capture against the SAC evader. This indicates that the LQR baseline is effective in simple cases, but its performance can drop significantly under more adversarial evasion behaviors. After introducing the RL residual term, the PPO+LQR pursuer policy shows much stronger adaptability to different evasion behaviors. As shown in Table 3, it achieves 100% capture against the LQR, SAC, and stationary evaders, and 99.8% capture against the PPO evader. Table 5 and Fig. 5 further show the results under different initial distances. The independent PPO policy still has almost no successful captures, while the LQR policy is sensitive to the evader type and initial distance. In contrast, PPO+LQR maintains near-perfect success against the PPO evader, with 100% at 20 and 25 km and 99.8% at 30 km. These results verify that the residual correction can significantly improve the capture capability of the pursuer.

Table 4 further shows that the PPO+LQR policy not only improves the mission success rate, but also outperforms the independent LQR policy in terms of average capture steps and cumulative maneuver cost in most cases. For example, against the LQR evader, the average successful capture steps are reduced from 25.26 to 18.30, and the average cumulative maneuver cost Cp(kf) is reduced from 85.53 to 70.96m/s. This result indicates that the PPO residual term does not improve performance by simply increasing the control magnitude. Instead, it refines the baseline LQR action while preserving the guidance provided by the prior strategy.

Fig. 4 presents the distribution of terminal episode steps when the evader adopts the LQR policy. The distributions show that the completion steps of the PPO+LQR policy are mainly concentrated before step 20, while the completion steps of the independent LQR policy are more scattered and extend to later steps. This concentration indicates that the PPO residual correction not only improves task completion, but also helps the pursuer complete the mission earlier. Fig. 5 further shows the initial-condition success maps under different initial distances. The pursuer model is trained with initial distances of 30–35km, and is then tested under different initial-distance settings. The independent PPO policy almost fails to capture the evader under all tested distances, which indicates that pure RL is difficult to train from scratch in this long-horizon and sparse-capture task. The independent LQR policy achieves partial success, but its successful and failed initial points are mixed, and its performance is sensitive to the initial distance. In contrast, the PPO+LQR policy maintains near-perfect capture rates across the tested initial distances. This result shows that the residual correction improves the robustness of the LQR baseline and helps the pursuer generalize to different initial-distance conditions. Fig. 6 shows a representative pursuit-evasion episode, including the 3D trajectory, relative distance, position variations, and per-step control components. The residual impulse is bounded by ±0.3m/s per axis, only 15% of the ±2.0m/s LQR bound. Therefore, LQR supplies the principal pursuit impulse and PPO provides bounded local corrections. These results verify the effectiveness of the proposed residual RL strategy.

4.3 Residual recurrent RL effectiveness validation

This subsection evaluates cross-opponent generalization. A single PPO-GRU+LQR pursuer is jointly trained against all evaders in the opponent pool. For comparison, PPO+LQR and PPO-GRU+LQR use the same pool, stage-wise sampling schedule, and remaining training settings.

The simulation settings are the same as those in the previous subsection. This opponent pool contains the evader policies described above. After training, each pursuer is tested separately against all four evader policies for 500 Monte Carlo episodes per pairing. The results are given in Tables 6 and 7 and Figs. 7 and 8.

Table 6 and Fig. 7 show that the PPO-GRU+LQR policy can handle different evader policies well and maintains a high capture success rate in all cases. Compared with the standard PPO+LQR policy, PPO-GRU+LQR improves the success rate against the LQR evader from 72.4% to 91.0%, and improves the success rate against the SAC evader from 93.2% to 96.0%. Against the PPO evader and the stationary evader, both methods maintain high success rates. This result indicates that the GRU module improves the generalization performance under multiple opponent strategies.

Table 7 shows that, although the PPO-GRU+LQR policy has better generalization ability, it is not always superior to the standard PPO+LQR policy in terms of average completion steps and cumulative maneuver cost. For example, against the LQR evader, PPO-GRU+LQR has a higher success rate, but its average successful steps and cumulative maneuver cost are slightly larger. Against the PPO, SAC, and stationary evaders, PPO-GRU+LQR achieves comparable or better average steps and cumulative maneuver cost. These results indicate that the main advantage brought by GRU lies in better robustness and stronger cross-opponent generalization, rather than absolute optimality in time or cumulative maneuver cost under every single scenario.

Fig. 8 shows a representative pursuit-evasion episode of PPO-GRU+LQR against the PPO evader, including the 3D trajectory, relative distance, position variations, and per-step control components. The action profiles show that, even after introducing the GRU structure, the PPO output still mainly acts as a correction to the baseline LQR action, rather than directly producing large maneuver commands. This indicates that the recurrent module mainly enhances temporal feature extraction, instead of changing the basic residual correction property of the proposed framework.

5 Conclusions

This paper proposes an R3LOOP framework for the three-dimensional orbital pursuit-evasion game. First, an LQR-based model prior is introduced to provide a nominal pursuit baseline for the pursuer. Then, a residual recurrent RL policy is constructed to generate corrections to the prior action and improve adaptability against different evader maneuvers. Finally, an opponent-pool training mechanism is developed to support policy training under diverse evader strategies. Simulation results verify the effectiveness of the proposed framework and demonstrate its advantages in capture performance and cross-opponent generalization ability. Note that the proposed framework is not limited to the specific combination of LQR and PPO. Instead, it provides a general hybrid example in which the model-based baseline strategy and the learning-based residual policy can be replaced by other suitable methods according to different task requirements. Future work will consider higher-fidelity orbital dynamics, imperfect observations, and more diverse opponent behaviors.

References

[1]

Yang K, Shen A, Xu N W. et al. A review of reinforcement learning approaches for pursuit-evasion games. Chinese Journal of Aeronautics, 2026, 39(6): 103940

[2]

Cao X Y, Ning X, Liu S Y. et al. Spacecraft intelligent orbital game technology: a review. Chinese Journal of Aeronautics, 2025, 38(6): 103480

[3]

Dong W, Wang C Y, Wang J N, et al. Spatial-temporal cooperative guidance for multiple flight vehicles. Singapore: Springer, 2025

[4]

Yi X, Wang C Y, Dong W. et al. Observation-shared cooperative look-angle tracking guidance for multiple missiles with partial seekers. IEEE Transactions on Aerospace and Electronic Systems, 2025, 61(5): 11586–11599

[5]

Di Mauro G, Lawn M, Bevilacqua R. Survey on guidance navigation and control requirements for spacecraft formation-flying missions. Journal of Guidance, Control, and Dynamics, 2018, 41(3): 581–602

[6]

Anderson G M, Grazier V W. Barrier in pursuit-evasion problems between two low-thrust orbital spacecraft. AIAA Journal, 1976, 14(2): 158–163

[7]

Jagat A, Sinclair A J. Nonlinear control for spacecraft pursuit-evasion game using the state-dependent Riccati equation method. IEEE Transactions on Aerospace and Electronic Systems, 2017, 53(6): 3032–3042

[8]

Han H Y, Dang Z H. Optimal delta-v-based strategies in orbital pursuit-evasion games. Advances in Space Research, 2023, 72(2): 243–256

[9]

Yang Q L, Luo Y Z, Guo X. et al. An approximate analytical method for solving the coplanar spacecraft pursuit-evasion game barrier. Acta Astronautica, 2026, 238: 717–730

[10]

Pontani M, Conway B A. Numerical solution of the three-dimensional orbital pursuit-evasion game. Journal of Guidance, Control, and Dynamics, 2009, 32(2): 474–487

[11]

Li Z Y, Zhu H, Yang Z. et al. Saddle point of orbital pursuit-evasion game under J2-perturbed dynamics. Journal of Guidance, Control, and Dynamics, 2020, 43(9): 1733–1739

[12]

Tang X, Ye D, Huang L. et al. Pursuit-evasion game switching strategies for spacecraft with incomplete-information. Aerospace Science and Technology, 2021, 119: 107112

[13]

Liu P X, Yang B, Li S. et al. Parameter-optimized pursuit strategy for orbital games with incomplete information. Journal of Guidance, Control, and Dynamics, 2025, 48(8): 1730–1742

[14]

Shi M M, Ye D, Sun Z W. et al. Spacecraft orbital pursuit-evasion games with J2 perturbations and direction-constrained thrust. Acta Astronautica, 2023, 202: 139–150

[15]

Chai Y, Luo J J, Wang M M. et al. Predictive pursuit-evasion game control method for approaching space non-cooperative target. IFAC-PapersOnLine, 2020, 53(2): 14882–14887

[16]

Liu Y C, Li C Y, Jiang J. et al. A model predictive Stackelberg solution to orbital pursuit-evasion game. Chinese Journal of Aeronautics, 2025, 38(2): 103198

[17]

Li Z Y, Luo Y Z. Orbital impulsive pursuit-evasion game formulation and Stackelberg equilibrium solutions. Journal of Spacecraft and Rockets, 2025, 62(2): 349–364

[18]

Shirobokov M, Trofimov S, Ovchinnikov M. Survey of machine learning techniques in spacecraft control design. Acta Astronautica, 2021, 186: 87–97

[19]

Federici L, Scorsoglio A, Zavoli A. et al. Meta-reinforcement learning for adaptive spacecraft guidance during finite-thrust rendezvous missions. Acta Astronautica, 2022, 201: 129–141

[20]

Li X, Wang X G. Online solution for orbital pursuit-evasion game via heterogeneous proximal policy optimization. IEEE Transactions on Aerospace and Electronic Systems, 2025, 61(5): 12044–12058

[21]

Gronauer S, Diepold K. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 2022, 55(2): 895–943

[22]

Zhang J C, Wang Y D, Zheng W. et al. Spacecraft pursuit-evasion game with incomplete information: a pre-training approximate dynamic programming method. Acta Astronautica, 2026, 243: 251–262

[23]

Zhang J R, Zhang K P, Zhang Y. et al. Near-optimal interception strategy for orbital pursuit-evasion using deep reinforcement learning. Acta Astronautica, 2022, 198: 9–25

[24]

Jiang R, Ye D, Xiao Y. et al. Orbital interception pursuit strategy for random evasion using deep reinforcement learning. Space: Science & Technology, 2023, 3: 0086

[25]

Zheng Z X, Zhang P, Yuan J P. Nonzero-sum pursuit-evasion game control for spacecraft systems: a Q-learning method. IEEE Transactions on Aerospace and Electronic Systems, 2023, 59(4): 3971–3981

[26]

Zhao L R, Zhang Y L, Dang Z H. PRD-MADDPG: an efficient learning-based algorithm for orbital pursuit-evasion game with impulsive maneuvers. Advances in Space Research, 2023, 72(2): 211–230

[27]

Yu W Z, Liu C, Yue X K. Reinforcement learning-based decision-making for spacecraft pursuit-evasion game in elliptical orbits. Control Engineering Practice, 2024, 153: 106072

[28]

Wang H B, Zhang Y. Impulsive maneuver strategy for multi-agent orbital pursuit-evasion game under sparse rewards. Aerospace Science and Technology, 2024, 155: 109618

[29]

Wang H B, Zhang Y, Liu H. et al. Impulsive thrust strategy for orbital pursuit-evasion games based on impulse-like constraint. Chinese Journal of Aeronautics, 2025, 38(1): 103180

[30]

Wei L S, Ning X, Lian X B. et al. Continuous-time hierarchical reinforcement learning for satellite pursuit decision. Chinese Journal of Aeronautics, 2025, 38(12): 103662

[31]

Staessens T, Lefebvre T, Crevecoeur G. Adaptive control of a mechatronic system using constrained residual reinforcement learning. IEEE Transactions on Industrial Electronics, 2022, 69(10): 10447–10456

[32]

Silver T, Allen K, Tenenbaum J, et al. Residual policy learning. arXiv preprint: arXiv: 1812.06298, 2018

[33]

Johannink T, Bahl S, Nair A, et al. Residual reinforcement learning for robot control. In: Proceedings of 2019 International Conference on Robotics and Automation (ICRA). Montreal, Canada, 2019, 6023–6029

[34]

Youm D, Jung H, Kim H. et al. Imitating and finetuning model predictive control for robust and symmetric quadrupedal locomotion. IEEE Robotics and Automation Letters, 2023, 8(11): 7799–7806

[35]

Cheng J, Kang D, Fadini G. et al. Rambo: RL-augmented model-based whole-body control for loco-manipulation. IEEE Robotics and Automation Letters, 2025, 10(9): 9462–9469

[36]

Holt H, Armellin R. Reinforcement learning enhanced LQR and control Lyapunov functions for spacecraft proximity operations. IEEE Transactions on Robotics, 2025, 41: 5117–5129

[37]

Clohessy W H, Wiltshire R S. Terminal guidance system for satellite rendezvous. Journal of the Aerospace Sciences, 1960, 27(9): 653–658

Rights & permissions

The Author(s). This article is published by Higher Education Press.

PDF (27669KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉