Multi-satellite cooperative scheduling for time-sensitive space targets tracking based on double deep Q-network

Abstract

Tracking time-sensitive space targets, characterized by high uncertainty and highly dynamic motion, requires multi-satellite coordination while accounting for the risk of target loss. This study proposes a mathematical model and a Time-Sensitive Space Multi-Target Observation Scheduling (TSSMTOS) algorithm that considers both tracking rewards and target loss scenarios. First, we establish a generalized real-time multi-satellite scheduling framework for tracking these unpredictable, rapidly evolving space targets. Subsequently, we introduce a Double Deep Q-network for Variable-Number Targets (DDQN-VNT). This approach enables the real-time allocation of dual satellites to each target, guided by heuristic rules, effectively addressing dynamic observation requirements. Experimental results demonstrate that DDQN-VNT outperforms traditional rule-based algorithms, the Parallel Dual Adaptive Genetic Algorithm (PDA-GA), and Adaptive Large Neighborhood Search (ALNS) in complex scenarios, exhibiting superior performance, enhanced efficiency, and robust generalization capabilities.

Graphical abstract

Keywords

multi-satellite scheduling / time-sensitive targets / double deep Q-network (DDQN) / reinforcement learning (RL)

References

[1] Bao X, Zhang S, Zhang X 2020. An effective method for satellite mission scheduling based on reinforcement learning. In: Proceedings of 2020 Chinese Automation Congress (CAC), Shanghai, China, 4037–4042

[2] Cao X, Li N, Qiu S, Li C, (2023). Research on the method of searching and tracking of the time-sensitive target through the mega-constellation. Aerospace Science and Technology, 137: 108299

[3] Chen M, Du Y, Tang K, Xing L, Chen Y, Chen Y, (2024). Learning to construct a solution for the agile satellite scheduling problem with time-dependent transition times. IEEE Transactions on Systems, Man, and Cybernetics. Systems, 54( 10): 5949–5963

[4] Chen M, Chun J, Pedrycz W, He Y, Liu X, Wu G, (2025). RCM: A neural policy model with reconstruction mechanism to construct a solution for the agile satellite scheduling problem. IEEE Transactions on Cybernetics, 55( 4): 1941–1953

[5] Cheng Y, Wei C, Sun S, You B, Zhao Y, (2023). An LEO constellation early warning system decision-making method based on hierarchical reinforcement learning. Sensors, 23( 4): 2225

[6] Chu X, Chen Y, Tan Y, (2017). An anytime branch and bound algorithm for agile earth observation satellite onboard scheduling. Advances in Space Research, 60( 9): 2077–2090

[7] Chun J, Chen M, Liu X, Xiang S, Du Y, Wu G, Xing L, (2024). Learning to construct a solution for UAV path planning problem with positioning error correction. Knowledge-Based Systems, 304: 112569

[8] Cui J, Zhang X, (2019). Application of a multi-satellite dynamic mission scheduling model based on mission priority in emergency response. Sensors, 19( 6): 1430

[9] Ferrari B, Cordeau J F, Delorme M, Iori M, Orosei R, (2025). Satellite scheduling problems: A survey of applications in earth and outer space observation. Computers & Operations Research, 173: 106875

[10] He L, Liu X, Laporte G, Chen Y, Chen Y, (2018). An improved adaptive large neighborhood search algorithm for multiple agile satellites scheduling. Computers & Operations Research, 100: 12–25

[11] Huang Y, Mu Z, Wu S, Cui B, Duan Y, (2021). Revising the observation satellite scheduling problem based on deep reinforcement learning. Remote Sensing, 13( 12): 2377

[12] Jiang X, Song Y, Xing L, (2022). Dual-population artificial bee colony algorithm for joint observation satellite mission planning problem. IEEE Access: Practical Innovations, Open Solutions, 10: 28911–28921

[13] Lemaître M, Verfaillie G, Jouhaud F, Lachiver J, Bataille N, (2002). Selecting and scheduling observations of agile satellites. Aerospace Science and Technology, 6( 5): 367–381

[14] Li B, Chen M, Xing L, Chen Y, Chen Y, (2025). Optimizing time-dependent multi-agile earth observation satellite scheduling problem using deep Q-learning and ensemble heuristics. Information Sciences, 712: 122140

[15] Li X, Chen J, Cai X, Cui N 2022. Satellite online scheduling algorithm based on proximal policy. In: Artificial Intelligence in China. Singapore: Springer, 100–108

[16] Li Z, Wang Y, Zheng W, Song X, (2023). Sensor scheduling with orbital uncertainty for space target tracking. Acta Astronautica, 205: 23–32

[17] Liu Y, Wen Z, Zhang S, Hu H, (2024). Learning-based constellation scheduling for time-sensitive space multi-target collaborative observation. Advances in Space Research, 73( 9): 4751–4766

[18] Liu Z, Xiong W 2021. A DQN-based hyperheuristic algorithm for emergency scheduling of earth observation satellites. In: Proceedings of 2021 2nd International Conference on Electronics, Communications and Information Technology, Sanya, China, 39–47

[19] Lu W, Gao W, Liu B, Niu W, Peng X, Yang Z, Song Y, (2024a). Parallel dual adaptive genetic algorithm: A method for satellite constellation task assignment in time-sensitive target tracking. Advances in Space Research, 74( 10): 5192–5213

[20] Lu W, Gao W, Liu B, Niu W, Wang D, Li Y, Peng X, Yang Z, (2024b). Reinforcement learning driven time-sensitive moving target tracking of intelligent agile satellite. IEEE Transactions on Aerospace and Electronic Systems, 60( 6): 9085–9101

[21] Shengyu Z, Zhencai Z, Haiying H, (2022). Burst tasks scheduling method for infrared LEO constellation based on multi-strategies. Journal of Beijing University of Aeronautics and Astronautics, 48( 12): 2405–2414

[22] Siew P M, Jang D, Roberts T G, Linares R, (2022). Space-based sensor tasking using deep reinforcement learning. Journal of the Astronautical Sciences, 69( 6): 1855–1892

[23] Song Y, Ou J, Pedrycz W, Suganthan P, Wang X, Xing L, Zhang Y, (2024a). Generalized model and deep reinforcement learning-based evolutionary method for multitype satellite observation scheduling. IEEE Transactions on Systems, Man, and Cybernetics. Systems, 54( 4): 2576–2589

[24] Sunehag P, Lever G, Gruslys A, Czarnecki W, Zambaldi V, Jaderberg M, Lanctot M, Sonnerat N, Leibo Joel Z, Tuyls K, Graepel T (2018). Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), Stockholm, Sweden, pp 2085–2087

[25] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez Aidan N, Kaiser L, Polosukhin I (2017). Attention is all you need. Preprint at arXiv. arXiv:1706.03762

[26] Wang H, Yang Z, Zhou W, Li D, (2019). Online scheduling of image satellites based on neural networks and deep reinforcement learning. Chinese Journal of Aeronautics, 32( 4): 1011–1019

[27] Wang P, Reinelt G, Gao P, Tan Y, (2011). A model, a heuristic and a decision support system to solve the scheduling problem of an earth observing satellite constellation. Computers & Industrial Engineering, 61( 2): 322–335

[28] Wang S (2012). Research on technology of initial alignment and inertial device calibration off the lunar surface. Dissertation for Master’s Degree. Harbin: Harbin Institute of Technology (in Chinese)

[29] Wang X, Wu J, Shi Z, Zhao F, Jin Z, (2022). Deep reinforcement learning-based autonomous mission planning method for high and low orbit multiple agile earth observing satellites. Advances in Space Research, 70( 11): 3478–3493

[30] Wei D, Guo C, Yang L, (2023). Intelligent hierarchical admission control for low-earth orbit satellites based on deep reinforcement learning. Sensors, 23( 20): 8470

[31] Wei L, Xing L, Wan Q, Song Y, Chen Y, (2021). A multi-objective memetic approach for time-dependent agile earth observation satellite scheduling problem. Computers & Industrial Engineering, 159: 107530

[32] Wen J (2022). Research on autonomous collaborative scheduling method of low-middle orbit satellite constellation for space moving target tracking. Dissertation for the Master’s Degree. Changsha: National University of Defense Technology (in Chinese)

[33] Yang H, Zhang Y, Bai X, Li S, (2024). Real-time satellite constellation scheduling for event-triggered cooperative tracking of space objects. IEEE Transactions on Aerospace and Electronic Systems, 60( 2): 2169–2182

[34] Yang W 2021. Research on multi-satellite collaborative planning and autonomous scheduling for moving targets tracking. Dissertation for the Doctoral Degree. Changsha: National University of Defense Technology (in Chinese)

[35] Zhang S (2021). Research on multi-target onboard cooperative observation technology for agile satellites. Dissertation for the Doctoral Degree. Shanghai: Innovation Academy for Microsatellites, Chinese Academy of Sciences (in Chinese)

[36] Zhang Y, Wang J, Yuan B, Wang C, Shi L, (2019). Research on multi-satellite observation multi-region task planning based on genetic algorithm. IOP Conference Series. Materials Science and Engineering, 685( 1): 012002

[37] Zhao X, Wang Z, Zheng G, (2020). Two-phase neural combinatorial optimization with reinforcement learning for agile satellite scheduling. Journal of Aerospace Information Systems, 17( 7): 346–357

[38] Zhou Z, Chen E, Wu F, Chang Z, Xing L, (2023). Multi-satellite scheduling problem with marginal decreasing imaging duration: An improved adaptive ant colony algorithm. Computers & Industrial Engineering, 176: 108890

RIGHTS & PERMISSIONS

Higher Education Press