Dogfight Simulation of Autonomous Swarm UAVs Based on Multi-Agent Deep Reinforcement Learning

Abstract

The operational utility of Unmanned Aerial Vehicles (UAVs) has evolved from passive surveillance to active engagement in disputed environments, where autonomous control must operate under highly dynamic and adversarial conditions. Hand-crafted heuristics often exhibit limited robustness when facing stochastic opponent behavior and non-stationary interactions. To address these challenges, we propose a Multi-Agent Deep Reinforcement Learning (MADRL) framework implemented in a Unity 6-based, physics-driven simulation that models flight dynamics and weapon kinematics. Agents are trained using Proximal Policy Optimization (PPO) with a composite reward function designed to encourage cooperative behaviors (e.g., coordinated target engagement) while enforcing safety constraints such as collision avoidance. In empirical evaluations, the learned policies achieve an 85% win rate against a heuristic baseline under the tested scenarios, exhibiting coordinated maneuvers and adaptive engagement strategies. These results indicate that multi-agent learning with decentralized execution can reduce operator workload and improve swarm effectiveness and survivability in conflict zone.

Keywords

Deep reinforcement learning (DRL) / Multi-agent systems (MAS) / Unmanned aerial vehicles (UAV) / Proximal policy optimization (PPO) / Autonomous combat simulation / Unity ML-agents

Statement of the Use of Generative AI and AI-Assisted Technologies in the Writing Process

During the preparation of this manuscript, the authors used ChatGPT 5.0 for translation, Google Gemini 3.1 Pro for improving grammatical clarity and Anthropic Claude Sonnet 4.5 for linguistic consistency checks. After using these services, the authors reviewed and edited the content as necessary and take full responsibility for the content of the published article.

Author Contributions

Conceptualization, E.B. and A.C.; Methodology, H.O.F.C. and A.C.; Software, H.O.F.C.; Validation, H.O.F.C., E.B. and A.C.; Formal Analysis, A.C.; Investigation, E.B.; Resources, E.B.; Data Curation, H.O.F.C.; Writing—Original Draft Preparation, E.B.; Writing—Review & Editing, A.C.; Visualization, H.O.F.C.; Supervision, A.C.

Ethics Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

All data generated or analyzed during this study are included in this published article. The simulation parameters, reward function specifications, and neural network hyperparameter configurations necessary to replicate the findings are detailed in Table 1 and Table 2.

Funding

This research received no external funding.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

[1] Austin R. Unmanned Aircraft Systems: UAVS Design, Development and Deployment; Wiley: Chichester, UK, 2010.

[2] Gupta H, Verma OP. Monitoring and Surveillance of Urban Road Traffic Using Low Altitude Drone Images: A Deep Learning Approach. Multimed. Tools Appl. 2022, 81, 19683-19703. DOI:10.1007/s11042-021-11146-x

[3] European Union Aviation Safety Agency (EASA). EASA and IATA Outline Comprehensive Plan to Mitigate GNSS Interference Risks. Press Release, 18 June 2025. Available online: https://www.easa.europa.eu/en/newsroom-and-events/press-releases/easa-and-iata-outline-comprehensive-plan-mitigate-gnss (accessed on 29 December 2025).

[4] Phadke A, Medrano FA. Towards resilient UAV swarms—A breakdown of resiliency requirements in UAV swarms. Drones 2022, 6, 340. DOI:10.3390/drones6110340

[5] Kong L, Wang L, Cao Z, Wang X. Resilience evaluation of UAV swarm considering resource supplementation. Reliab. Eng. Syst. Saf. 2024, 241, 109673. DOI:10.1016/j.ress.2023.109673

[6] Rowe LI, Hattie J, Munro J, Khan IA. High-performing teams: Is collective intelligence the answer? PLoS ONE 2024, 19, e0307945. DOI:10.1371/journal.pone.0307945

[7] Dorri A, Kanhere SS, Jurdak R. Multi-Agent Systems: A Survey. IEEE Access 2018, 6, 28573-28593. DOI:10.1109/ACCESS.2018.2831228

[8] Shi L, Yan S, Li W. Consensus and products of substochastic matrices: Convergence rate with communication delays. IEEE Trans. Syst. Man Cybern. Syst. 2025, 55, 4752-4761. DOI:10.1109/TSMC.2025.3559667

[9] Bu Y, Yan Y, Yang Y. Advancement challenges in UAV swarm formation control: A comprehensive review. Drones 2024, 8, 320. DOI:10.3390/drones8070320

[10] Li W, Yan S, Shi L, Yue J, Shi M, Lin B, et al. Multiagent Consensus Tracking Control Over Asynchronous Cooperation-Competition Networks. IEEE Trans. Cybern. 2025, 55, 4347-4360. DOI:10.1109/TCYB.2025.3583387

[11] Drew DS. Multi-Agent Systems for Search and Rescue Applications. Curr. Robot. Rep. 2021, 2, 189-200. DOI:10.1007/s43154-021-00048-3

[12] Xiao J, Wang G, Zhang Y, Cheng L. A Distributed Multi-Agent Dynamic Area Coverage Algorithm Based on Reinforcement Learning. IEEE Access 2020, 8, 33511-33521. DOI:10.1109/ACCESS.2020.2967225

[13] Ekechi CC, Elfouly T, Alouani A, Khattab T. A survey on UAV control with multi-agent reinforcement learning. Drones 2025, 9, 484. DOI:10.3390/drones9070484

[14] Sutton RS, Barto AG. Reinforcement Learning: An Introduction; MIT Press: Cambridge, UK, 1998; Volume 1, pp. 9-11.

[15] Ning Z, Xie L. A survey on multi-agent reinforcement learning and its application. J. Autom. Intell. 2024, 3, 73-91. DOI:10.1016/j.jai.2024.02.003

[16] Li T, Shi D, Jin S, Wang Z, Yang H, Chen Y. Multi-agent hierarchical graph attention actor-critic reinforcement learning. Entropy 2024, 27, 4. DOI:10.3390/e27010004

[17] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. DOI:10.48550/arXiv.1707.06347

[18] Ghasemi M, Moosavi AH, Ebrahimi D. A comprehensive survey of reinforcement learning: From algorithms to practical challenges. arXiv 2024, arXiv:2411.18892. DOI:10.48550/arXiv.2411.18892

[19] Watkins CJ, Dayan P. Q-learning. Mach. Learn. 1992, 8, 279-292. DOI: 10.1007/BF00992698

[20] Ye L, Li R, Hu X, Li J, Xing B, Peng Y, et al. Unity RL Playground: A Versatile Reinforcement Learning Framework for Mobile Robots. arXiv 2025, arXiv:2503.05146. DOI:10.48550/arXiv.2503.05146

[21] Tan CB, Toledo E, Ellis B, Foerster JN, Huszár F. Beyond the Boundaries of Proximal Policy Optimization. arXiv 2024, arXiv:2411.00666. DOI:10.48550/arXiv.2411.00666

[22] Lanham M. Learn Unity ML-Agents-Fundamentals of Unity Machine Learning: Incorporate New Powerful ML Algorithms Such as Deep Reinforcement Learning for Games; Packt Publishing Ltd.: Birmingham, UK, 2018. ISBN 978-1-78913-813-9

[23] Chen C, Song T, Mo L, Lv M, Lin D. Autonomous dogfight decision-making for air combat based on reinforcement learning with automatic opponent sampling. Aerospace 2025, 12, 265. DOI:10.3390/aerospace12030265

[24] Zhu J, Kuang M, Zhou W, Shi H, Zhu J, Han X. Mastering air combat game with deep reinforcement learning. Def. Technol. 2024, 34, 295-312. DOI:10.1016/j.dt.2023.08.019