Multi-agent deep deterministic policy gradient algorithm for predictive UAV deployment in CF-MIMO for identifying coverage holes

Abstract

Unmanned aerial vehicles, due to their adaptable mobility and various applications, including supporting communication infrastructure, monitoring, and rescue, are becoming increasingly valuable, making them a valuable addition to emergency communication networks. Even though cell-free massive multiple input multiple outputs (CF-mMIMO) networks provide high communication data rates, their immobility makes it difficult to maintain quality network continuity in emergency, unpredictable, and congested areas where users’ equipment is located. To mitigate this challenge, the integration of aerial access points (AAPs) into CF-mMIMO networks is proposed by using the multi-agent deep deterministic policy gradient (MADDPG) framework, which teaches several unmanned aerial vehicles (UAVs) to jointly learn the best deployment plans by estimating user distributions and traffic demand trends on invitations to provide tremendous dynamic coverage, increased spectral efficiency, and throughput maximization. The predictive component framework utilizes a long short-term memory (LSTM) network model incorporating concepts of learning, association, movement, and service provision for temporal traffic forecasting, thereby ensuring proactive UAV positioning before coverage holes emerge. Our extensive simulation results demonstrate that the MADDPG-based throughput deployment strategy achieves approximately 45 Gb/s for 50 UAVs, spectral efficiency for downlink and uplink of 10.2 bps/Hz, 15.2 bps/Hz, respectively, and minimal transmit power of 3.5 kJ as compared with the multi-agent soft actor-critic (MASAC) method, traditional heuristic-LSTM, and single-agent reinforcement learning approaches.

Keywords

Aerial access points / Cell-free massive multiple input multiple outputs / Long short-term memory / Multi-agent deep deterministic policy gradient / Unmanned aerial vehicles

CRediT authorship contribution statement

Kenneth Okello contributed completely to writing the original research draft, installation of machine learning software, Methodology, modeling, coding, parameter tuning, investigation, visualization, and validation. Elijah Mwangi He contributed more to research, Supervision, Writing, review & editing, methodology concept analysis, Visualization, and Validation. Dominic Bernard Onyango Konditi He also contributed more to research Supervision, Writing, review & editing, methodology concept analysis, Visualization, and Validation. All authors have read and agreed to the published version of the manuscript.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

We are grateful to Pan University, Institute for Basic Sciences Technology and Innovation (PAUSTI), and our colleagues for their valuable feedback and constructive discussions that enriched this study.

References

[1] M. Pons, E. Valenzuela, B. Rodríguez, et al., Utilization of 5G technologies in IoT applications: current limitations by interference and network optimization difficulties—a review, Sensors, 2 3 (8) (2023) p. 3876.

[2] G.Y. Liu, Y.H. Huang, N. Li, et al., Vision, requirements and network architecture of 6G mobile network beyond 2030, China Commun, 1 7 (9) (2020) pp. 92-104.

[3] N.S. Rani, D.V. Vardhan, G. Mamatha, et al., Cell-free massive MIMO versus small cells, In: Proc. of the 2nd Intl. Conf. on Cognitive and Intelligent Computing, Hyderabad, India, 2023, pp. 23-32.

[4] D. Diarra, H.O. Absaloms, P.K. Langat, Deep reinforcement learning-based power allocation for maximizing energy efficiency in mmWave cell-free massive MIMO, Int. J. Intell. Eng. Syst., 1 8 (10) (2025) pp. 450-465.

[5] O. Kenneth, E. Mwangi, D.B.O. Konditi, Optimizing UAV location for deployment in cell-free massive MIMO networks using a soft actor-critic reinforcement learning, IEEE Access, 1 3 (2025) pp. 139680-139695, https://doi.org/10.1109/ACCESS.2025.3592733.

[6] X.J. Liu, X.W. Wang, M. Huang, et al., Deployment of UAV-BSs for on-demand full communication coverage, Ad Hoc Netw 140 (2023) 103047, https://doi.org/10.1016/j.adhoc.2022.103047.

[7] C.Y. Pan, J.C. Wang, X.W. Yue, et al., Joint AP selection and task offloading based on deep reinforcement learning for urban-micro cell-free UAV network, Electronics-Switz, 1 2 (23) (2023) p. 4777.

[8] C.W. Wang, D.H. Deng, L.X. Xu, W.D. Wang, Resource scheduling based on deep reinforcement learning in UAV assisted emergency communication networks, IEEE T. Commun., 7 0 (6) (2022) pp. 3834-3848.

[9] A.F. Mostafa, M. Abdel-Kader, Y. Gadallah, et al., Machine learning-based multi-UAV deployment for uplink traffic sizing and offloading in cellular networks, IEEE Access, 1 1 (2023) pp. 71314-71325, https://doi.org/10.1109/ACCESS.2023.3293148.

[10] X.L. Luo, C. Chen, C.N. Zeng, et al., Deep reinforcement learning for joint trajectory planning, transmission scheduling, and access control in UAV-assisted wireless sensor networks, Sensors, 2 3 (10) (2023) p. 4691.

[11] A. Manzoor, M. Ozger, C. Cavdar, Optimizing UAV deployment for enhanced cellular network capacity: a joint optimization approach, roc. of the 6th Intl. Conf. on Communications, Signal Processing, and their Applications, Istanbul, Turkiye, 2024, pp. 1-6.

[12] G.B. Tarekegn, R.T. Juang, B.A. Tesfaw, et al., A centralized multi-agent DRL-based trajectory control strategy for unmanned aerial vehicle-enabled wireless communications, IEEE Open J. Veh. Tech., 5 (2024) pp. 1230-1241, https://doi.org/10.1109/OJVT.2024.3451143.

[13] L. Tsipi, V.I. Tatsis, D.N. Skoutas, et al., A machine learning UAV deployment approach for emergency cellular networks, In: Proc. of IEEE International Conference on Communications, Rome, Italy, 2023, pp. 5683-5688.

[14] Y. Yang, Y.J. Shi, X. Cui, et al., A hybrid decision-making framework for UAV-assisted MEC systems: integrating a dynamic adaptive genetic optimization algorithm and soft actor–critic algorithm with hierarchical action decomposition and uncertainty-quantified critic ensemble, Drones, 9 (3) (2025) p. 206.

[15] K. Hassan, M. Masarra, M. Zwingelstein, et al., Channel estimation techniques for millimeter-wave communication systems: achievements and challenges, IEEE Open J. Commun. Soc., 1 (2020) pp. 1336-1363, https://doi.org/10.1109/OJCOMS.2020.3015394.

[16] S. Mukherjee, R. Chopra, Performance analysis of cell-free massive MIMO systems in LoS/NLoS channels, IEEE T. Veh. Technol., 7 1 (6) (2022) pp. 6410-6423.

[17] A. Saboor, Z.Z. Cui, E. Vinogradov, et al., Path loss modelling for UAV communications in urban scenarios with random obstacles, [Online] January 2025. Available, https://arxiv.org/abs/2501.14411.

[18] A. Al-Hourani, S. Kandeepan, S. Lardner, Optimal LAP altitude for maximum coverage, IEEE Wirel. Commun, Le, 3 (6) (2014) pp. 569-572.

[19] A. Saboor, E. Vinogradov, Z.Z. Cui, et al., Probability of line of sight evaluation in urban environments using 3D simulator, In: Proc. of IEEE Intl. Black Sea Conference on Communications and Networking, Istanbul, Turkiye, 2023, pp. 135-140.

[20] H.Q. Gu, Q. Zhang, Z.W. Chai, et al., Cell-free massive MIMO with limited-capacity backhaul: power control and quantization optimization, Phys. Commun.-Amst. 71 (2025) 102652, https://doi.org/10.1016/j.phycom.2025.102652.

[21] E. Nayebi, TDD massive MIMO systems: channel estimation, power optimization, and access point location design, [Online]. Available,https://escholarship.org/uc/item/8843p6d8, 2018).

[22] O.S. Oubbati, J. Alotaibi, F. Alromithy, et al., A UAV-UGV cooperative system: patrolling and energy management for urban monitoring, IEEE T. Veh. Technol., 7 4 (9) (2025) pp. 13521-13536.

[23] N. Gao, Y. Zeng, J. Wang, et al., Energy model for UAV communications: experimental validation and model generalization, China Commun., 1 8 (7) (2021) pp. 253-264.

[24] X.-T. Dang, J.-S. Eom, B.-M. Vu, et al., A DDPG-LSTM framework for optimizing UAV-enabled integrated sensing and communication, Drones, 9 (8) (2025) p. 548.

[25] A.M. Shalaby, N.S. Othman, The effect of rainfall on the UAV placement for 5G spectrum in Malaysia, Electronics, 1 1 (5) (2022) p. 681.

[26] D.T. Hoang, N. Van Huynh, D.N. Nguyen, et al., Markov decision process and reinforcement learning, D.T. Hoang, N. Van Huynh, D.N. Nguyen, E. Hossain, D. Niyato (Eds.), Deep Reinforcement Learning for Wireless Communications and Networking: Theory, Applications and Implementation, Wiley, New York, USA, 2023, pp. 25-36.

[27] J. Alotaibi, O.S. Oubbati, M. Atiquzzaman, et al., Optimizing disaster response with UAV-mounted RIS and HAP-enabled edge computing in 6G networks, J. Netw. Comput. Appl. 241 (2025) 104213, https://doi.org/10.1016/j.jnca.2025.104213.

[28] Z.X. Zhu, N.H. Xie, K. Zong, et al., Building a connected communication network for UAV clusters using DE-MADDPG, Symmetry-Basel, 1 3 (8) (2021) p. 1537.

[29] O.S. Oubbati, N. Chaib, A. Lakas, et al., On-demand routing for urban VANETs using cooperating UAVs, Communications in Network Technologies, El Oued, Algeria, 2018, pp. 108-113.