RS-DRL-based offloading policy and UAV trajectory design in F-MEC systems

Abstract

For better flexibility and greater coverage areas, Unmanned Aerial Vehicles (UAVs) have been applied in Flying Mobile Edge Computing (F-MEC) systems to offer offloading services for the User Equipment (UEs). This paper considers a disaster-affected scenario where UAVs undertake the role of MEC servers to provide computing resources for Disaster Relief Devices (DRDs). Considering the fairness of DRDs, a max-min problem is formulated to optimize the saved time by jointly designing the trajectory of the UAVs, the offloading policy and serving time under the constraint of the UAVs' energy capacity. To solve the above non-convex problem, we first model the service process as a Markov Decision Process (MDP) with the Reward Shaping (RS) technique, and then propose a Deep Reinforcement Learning (DRL) based algorithm to find the optimal solution for the MDP. Simulations show that the proposed RS-DRL algorithm is valid and effective, and has better performance than the baseline algorithms.

Keywords

Flying mobile edge computing / Task offloading / Reward shaping / Deep reinforcement learning

CRediT authorship contribution statement

Yulu Yang: Conceptualization, Data curation, Formal analysis, Methodology, Software, Validation, Visualization, Writing - original draft, Writing - review & editing. Han Xu: Conceptualization, Formal analysis, Methodology, Project administration. Zhu Jin: Writing - review & editing. Tiecheng Song: Conceptualization, Funding acquisition, Project administration, Resources, Supervision. Jing Hu: Funding acquisition, Project administration, Resources, Supervision. Xiaoqin Song: Project administration, Supervision, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported by the Key Research and Development Program of Jiangsu Province (No. BE2020084-2) and the National Key Research and Development Program of China (No. 2020YFB1600104).

References

[1] R. Apanaviciene, A. Vanagas, P.A. Fokaides, Smart building integration into a smart city (SBISC): development of a new evaluation framework, Energies 13 (9) (2020) 2190.

[2] Y. Yin, G. Xu, P. Zhu, X. Yin, A 5G-enabled and self-powered sensor data man-agement scheme for the smart medical platform system, IEEE Sens. J. (2022) 20904-20915.

[3] L. P, G. Rejith, T. Toby, S.S. N. B., S. N. Rao,A resilient IoT system architecture for disaster management in collapsed buildings, in:2022 International Conference on Wireless Communications Signal Processing and Networking (WiSPNET), 2022, pp. 282-287.

[4] E. Nagaoka, R. Banno, O. Mizuno,Implementation and node load evaluation of cloud-based load distribution model in ICSN, in:2022 23rd Asia-Pacific Network Operations and Management Symposium (APNOMS), 2022, pp. 1-4.

[5] Y. Li, S. Xia, M. Zheng, B. Cao, Q. Liu, Lyapunov optimization-based trade-off policy for mobile cloud offloading in heterogeneous wireless networks, IEEE Trans. Cloud Comput. 10 (1) (2019) 491-505.

[6] C. Park, J. Lee, Mobile edge computing-enabled heterogeneous networks, IEEE Trans. Wirel. Commun. 20 (2) (2021) 1038-1051.

[7] Y. Mao, C. You, J. Zhang, K. Huang, K.B. Letaief, A survey on mobile edge com-puting: the communication perspective, IEEE Commun. Surv. Tutor. 19 (4) (2017) 2322-2358.

[8] H. Tran-Dang, D.-S. Kim, A survey on matching theory for distributed computa-tion offloading in IoT-fog-cloud systems: perspectives and open issues, IEEE Access (2022) 118353-118369.

[9] Y. Li, H. Ma, L. Wang, S. Mao, G. Wang, Optimized content caching and user associ-ation for edge computing in densely deployed heterogeneous networks, IEEE Trans. Mob. Comput. 21 (6) (2020) 2130-2142.

[10] F. Cheng, G. Gui, N. Zhao, Y. Chen, J. Tang, H. Sari, UAV-relaying-assisted secure transmission with caching, IEEE Trans. Commun. 67 (5) (2019) 3140-3153.

[11] Y. Liu, J. Zhou, D. Tian, Z. Sheng, X. Duan, G. Qu, V.C.M. Leung, Joint com-munication and computation resource scheduling of a UAV-assisted mobile edge computing system for platooning vehicles, IEEE Trans. Intell. Transp. Syst. 23 (7) (2022) 8435-8450.

[12] S. Xia, Z. Yao, Y. Li, S. Mao, Online distributed offloading and computing re-source management with energy harvesting for heterogeneous MEC-enabled IoT, IEEE Trans. Wirel. Commun. 20 (10) (2021) 6743-6757.

[13] X. Cao, J. Xu, R. Zhang,Mobile edge computing for cellular-connected UAV: com-putation offloading and trajectory optimization, in: 2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), 2018, pp. 1-5.

[14] Z. Li, P. Tong, J. Liu, X. Wang, L. Xie, H. Dai, Learning-based data gathering for in-formation freshness in UAV-assisted IoT networks, IEEE Int. Things J. 10 (3) (2022) 2557-2573.

[15] F. Machida, E. Andrade,PA-Offload: performability-aware adaptive fog offloading for drone image processing, in: 2021 IEEE 5th International Conference on Fog and Edge Computing (ICFEC), 2021, pp. 66-73.

[16] Q. Li, L. Shi, Z. Zhang, G. Zheng, Resource allocation in UAV-enabled wireless powered MEC networks with hybrid passive and active communications, IEEE Int. Things J. 10 (3) (2022) 2574-2588.

[17] Y. Liu, K. Xiong, Q. Ni, P. Fan, K.B. Letaief, UAV-assisted wireless powered co-operative mobile edge computing: joint offloading, CPU control, and trajectory optimization, IEEE Int. Things J. 7 (4) (2020) 2777-2790.

[18] Y. Du, K. Wang, K. Yang, G. Zhang, Energy-efficient resource allocation in UAV based MEC system for IoT devices, in: 2018 IEEE Global Communications Confer-ence (GLOBECOM), IEEE, 2018, pp. 1-6.

[19] M. Li, N. Cheng, J. Gao, Y. Wang, L. Zhao, X. Shen, Energy-efficient UAV-assisted mobile edge computing: resource allocation and trajectory optimization, IEEE Trans. Veh. Technol. 69 (3) (2020) 3424-3438.

[20] Y.K. Tun, Y.M. Park, N.H. Tran, W. Saad, S.R. Pandey, C.S. Hong, Energy-efficient resource management in UAV-assisted mobile edge computing, IEEE Commun. Lett. 25 (1) (2021) 249-253.

[21] J. Ji, K. Zhu, C. Yi, D. Niyato, Energy consumption minimization in UAV-assisted mobile-edge computing systems: joint resource allocation and trajectory design, IEEE Int. Things J. 8 (10) (2021) 8570-8584.

[22] I. Budhiraja, N. Kumar, S. Tyagi, S. Tanwar, Energy consumption minimization scheme for NOMA-based mobile edge computation networks underlaying UAV, IEEE Syst. J. 15 (4) (2021) 5724-5733.

[23] H. He, S. Zhang, Y. Zeng, R. Zhang, Joint altitude and beamwidth optimization for UAV-enabled multiuser communications, IEEE Commun. Lett. 22 (2) (2018) 344-347.

[24] W. Pi, J. Zhou, Multi-UAV enabled data collection with efficient joint adaptive in-terference management and trajectory design, Electronics 10 (5) (2021) 547.

[25] C. Zhan, H. Hu, X. Sui, Z. Liu, D. Niyato, Completion time and energy optimization in the UAV-enabled mobile-edge computing system, IEEE Int. Things J. 7 (8) (2020) 7808-7822.

[26] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, A. Nallanathan, Deep reinforcement learning based dynamic trajectory control for UAV-assisted mobile edge computing, IEEE Trans. Mob. Comput. 21 (10) (2022) 3536-3550.

[27] Z. Wu, Z. Yang, C. Yang, J. Lin, Y. Liu, X. Chen, Joint deployment and trajectory op-timization in UAV-assisted vehicular edge computing networks, J. Commun. Netw. 24 (1) (2022) 47-58.

[28] A.M. Seid, G.O. Boateng, S. Anokye, T. Kwantwi, G. Sun, G. Liu, Collaborative com-putation offloading and resource allocation in multi-UAV-assisted IoT networks: a deep reinforcement learning approach, IEEE Int. Things J. 8 (15) (2021) 16.

[29] Y. Liu, J. Yan, X. Zhao, Deep reinforcement learning based latency minimization for mobile edge computing with virtualization in maritime UAV, communication network, IEEE Trans. Veh. Technol. 71 (4) (2022) 4225-4236.

[30] N. Zhao, Z. Ye, Y. Pei, Y.-C. Liang, D. Niyato, Multi-agent deep reinforcement learn-ing for task offloading in UAV-assisted mobile edge computing, IEEE Trans. Wirel. Commun. 21 (9) (2022) 6949-6960.

[31] S. Xia, Z. Yao, G. Wu, Y. Li, Distributed offloading for cooperative intelligent trans-portation under heterogeneous networks, IEEE Trans. Intell. Transp. Syst. 23 (9) (2022) 16701-16714.

[32] J. Li, Q. Liu, P. Wu, F. Shu, S. Jin,Task offloading for UAV-based mobile edge com-puting via deep reinforcement learning, in:2018 IEEE/CIC International Conference on Communications in China (ICCC), 2018, pp. 798-802.

[33] Q. Liu, L. Shi, L. Sun, J. Li, M. Ding, F. Shu, Path planning for UAV-mounted mobile edge computing with deep reinforcement learning, IEEE Trans. Veh. Technol. 69 (5) (2020) 5723-5728.

[34] C. Zhan, H. Hu, Z. Liu, Z. Wang, S. Mao, Multi-UAV-enabled mobile-edge com-puting for time-constrained IoT applications, IEEE Int. Things J. 8 (20) (2021) 15553-15567.

[35] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, L. Hanzo, Multi-agent deep rein-forcement learning-based trajectory planning for multi-UAV assisted mobile edge computing, IEEE Trans. Cogn. Commun. Netw. 7 (1) (2021) 73-84.

[36] Z. Qin, Z. Wei, Y. Qu, F. Zhou, H. Wang, D.W.K. Ng, C.-B. Chae, AoI-aware schedul-ing for air-ground collaborative mobile edge computing, IEEE Trans. Wirel. Com-mun. (2022) 2989-3005.

[37] A. Al-Hourani, S. Kandeepan, S. Lardner, Optimal LAP altitude for maximum cover-age, IEEE Wirel. Commun. Lett. 3 (6) (2014) 569-572.

[38] R.S. Sutton, A.G. Barto, Reinforcement Learning:An Introduction, MIT Press, 2018.

[39] A.Y. Ng, D. Harada, S. Russell, Policy invariance under reward transformations: theory and application to reward shaping, in: Icml, 1999, pp. 278-287.

[40] W.W. Bian, J. Wei, K.H. Huang, J.X. Wang, X. Lv, W.N. Yuan,Intelligent decision al-gorithm of target compound interception based on A2C-PPO, in:2021 International Conference on Cyber-Physical Social Intelligence (ICCSI), 2021, pp. 1-6.

[41] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy opti-mization algorithms, arXiv preprint, arXiv :1707.06347.