Trajectory optimization for UAV-enabled relaying with reinforcement learning

Abstract

In this paper, we investigate the application of the Unmanned Aerial Vehicle (UAV)-enabled relaying system in emergency communications, where one UAV is applied as a relay to help transmit information from ground users to a Base Station (BS). We maximize the total transmitted data from the users to the BS, by optimizing the user communication scheduling and association along with the power allocation and the trajectory of the UAV. To solve this non-convex optimization problem, we propose the traditional Convex Optimization (CO) and the Reinforcement Learning (RL)-based approaches. Specifically, we apply the block coordinate descent and successive convex approximation techniques in the CO approach, while applying the soft actor-critic algorithm in the RL approach. The simulation results show that both approaches can solve the proposed optimization problem and obtain good results. Moreover, the RL approach establishes emergency communications more rapidly than the CO approach once the training process has been completed.

Keywords

Unmanned aerial vehicle / Emergency communications / Trajectory optimization / Convex optimization / Reinforcement learning

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported in part by the Shenzhen Basic Research Project under Grant JCYJ20220531103008018 and Grant 20200812112423002, in part by the Guangdong Basic Research Program under Grant 2019A1515110358, 2021A1515012097, in part by the open research fund of National Mobile Communications Research Laboratory, Southeast University (No. 2021D16).

References

[1] Y. Zeng, R. Zhang, T.J. Lim, Wireless communications with unmanned aerial vehi-cles: opportunities and challenges, IEEE Commun. Mag. 54 (5) (2016) 36-42.

[2] W. Feng, J. Tang, Y. Yu, J. Song, N. Zhao, G. Chen, K. Wong, J. Chambers, UAV-enabled SWIPT in IoT networks for emergency communications, IEEE Wirel. Com-mun. 27 (5) (2020) 140-147.

[3] M. Mozaffari, W. Saad, M. Bennis, M. Debbah, Unmanned aerial vehicle with un-derlaid device-to-device communications: performance and tradeoffs, IEEE Trans. Wirel. Commun. 15 (6) (2016) 3949-3963.

[4] X. Pang, J. Tang, N. Zhao, X. Zhang, Y. Qian, Energy-efficient design for mmwave-enabled noma-uav networks, Sci. China Inf. Sci. 64 (4) (2021) 1-14.

[5] C. Pan, H. Ren, Y. Deng, M. Elkashlan, A. Nallanathan, Joint blocklength and loca-tion optimization for URLLC-enabled UAV relay systems, IEEE Commun. Lett. 23 (3) (2019) 498-501.

[6] L. Xiao, Y. Xu, D. Yang, Y. Zeng, Secrecy energy efficiency maximization for UAV-enabled mobile relaying, IEEE Trans. Green Commun. Netw. 4 (1) (2020) 180-193.

[7] P. Yang, X. Cao, C. Yin, Z. Xiao, X. Xi, D. Wu, Proactive drone-cell deployment: overload relief for a cellular network under flash crowd traffic, IEEE Trans. Intell. Transp. Syst. 18 (10) (2017) 2877-2892.

[8] J. Zhang, Y. Zeng, R. Zhang, Spectrum and energy efficiency maximization in UAV-enabled mobile relaying, in: IEEE International Conference on Communications (ICC), Paris, France, 2017, pp. 1-6.

[9] J. Zhao, Q. Li, Y. Gong, K. Zhang, Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks, IEEE Trans. Veh. Technol. 68 (8) (2019) 7944-7956.

[10] Y. Zhao, R. Adve, T.J. Lim, Improving amplify-and-forward relay networks: optimal power allocation versus selection, in: IEEE International Symposium on Information Theory, Seattle, WA, USA, 2006, pp. 1234-1238.

[11] J. Laneman, D. Tse, G. Wornell, Cooperative diversity in wireless networks: efficient protocols and outage behavior, IEEE Trans. Inf. Theory 50 (12) (2004) 3062-3080.

[12] A. Sendonaris, E. Erkip, B. Aazhang, User cooperation diversity. Part II. Imple-mentation aspects and performance analysis, IEEE Trans. Commun. 51 (11) (2003) 1939-1948.

[13] K. Zhou, T.M. Lok, Optimal power allocation for relayed transmission through a mobile relay node, in: IEEE 71st Vehicular Technology Conference, Taipei, Taiwan, 2010, pp. 1-5.

[14] Y. Zeng, R. Zhang, T.J. Lim, Throughput maximization for UAV-enabled mobile relaying systems, IEEE Trans. Commun. 64 (12) (2016) 4983-4996.

[15] M. Erdelj, E. Natalizio, K.R. Chowdhury, I.F. Akyildiz, Help from the sky: leveraging UAVs for disaster management, IEEE Pervasive Comput. 16 (1) (2017) 24-32.

[16] N. Zhao, W. Lu, M. Sheng, Y. Chen, J. Tang, F.R. Yu, K.-K. Wong, UAV-assisted emergency networks in disasters, IEEE Wirel. Commun. 26 (1) (2019) 45-51.

[17] P. Li, J. Xu, Fundamental rate limits of UAV-enabled multiple access channel with trajectory optimization, IEEE Trans. Wirel. Commun. 19 (1) (2020) 458-474.

[18] J. Xu, Y. Zeng, R. Zhang, UAV-enabled wireless power transfer: trajectory design and energy optimization, IEEE Trans. Wirel. Commun. 17 (8) (2018) 5092-5106.

[19] J. Cui, Y. Liu, A. Nallanathan, Multi-agent reinforcement learning-based resource allocation for UAV networks, IEEE Trans. Wirel. Commun. 19 (2) (2020) 729-743.

[20] Y. Zeng, J. Xu, R. Zhang, Energy minimization for wireless communication with rotary-wing UAV, IEEE Trans. Wirel. Commun. 18 (4) (2019) 2329-2345.

[21] Q. Wu, Y. Zeng, R. Zhang, Joint trajectory and communication design for multi-UAV enabled wireless networks, IEEE Trans. Wirel. Commun. 17 (3) (2018) 2109-2121.

[22] S. Fang, G. Chen, Y. Li, Joint optimization for secure intelligent reflecting surface assisted UAV networks, IEEE Wirel. Commun. Lett. 10 (2) (2021) 276-280.

[23] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, A. Nallanathan, Deep reinforcement learning based dynamic trajectory control for UAV-assisted mobile edge computing, IEEE Trans. Mob. Comput. 21 (10) (2022) 3536-3550.

[24] S. Yin, S. Zhao, Y. Zhao, F.R. Yu, Intelligent trajectory design in UAV-aided com-munications with reinforcement learning, IEEE Trans. Veh. Technol. 68 (8) (2019) 8227-8231.

[25] C.H. Liu, Z. Chen, J. Tang, J. Xu, C. Piao, Energy-efficient UAV control for effective and fair communication coverage: a deep reinforcement learning approach, IEEE J. Sel. Areas Commun. 36 (9) (2018) 2059-2070.

[26] L. Xiao, X. Lu, D. Xu, Y. Tang, L. Wang, W. Zhuang, UAV relay in VANETs against smart jamming with reinforcement learning, IEEE Trans. Veh. Technol. 67 (5) (2018) 4087-4097.

[27] U. Challita, W. Saad, C. Bettstetter, Interference management for cellular-connected UAVs: a deep reinforcement learning approach, IEEE Trans. Wirel. Commun. 18 (4) (2019) 2125-2140.

[28] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, L. Hanzo, Multi-agent deep rein-forcement learning-based trajectory planning for multi-UAV assisted mobile edge computing, IEEE Trans. Cogn. Commun. Netw. 7 (1) (2021) 73-84.

[29] A.S. Khan, G. Chen, Y. Rahulamathavan, G. Zheng, B. Assadhan, S. Lambotharan, Trusted UAV network coverage using blockchain, machine learning, and auction mechanisms, IEEE Access 8 (2020) 118219-118234.

[30] Y. He, Z. Zhang, F.R. Yu, N. Zhao, H. Yin, V.C.M. Leung, Y. Zhang, Deep-reinforcement-learning-based optimization for cache-enabled opportunistic inter-ference alignment wireless networks, IEEE Trans. Veh. Technol. 66 (11) (2017) 10433-10445.

[31] A. Kwasinski, W.W. Weaver, P.L. Chapman, P.T. Krein, Telecommunications power plant damage assessment for hurricane Katrina-Site survey and follow-up results, IEEE Syst. J. 3 (3) (2009) 277-287.

[32] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: off-policy maximum en-tropy deep reinforcement learning with a stochastic actor, arXiv :1801.01290, 2018.

[33] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, S. Levine,Soft actor-critic algorithms and applications, arXiv : 1812.05905, 2018.