Partial observation learning-based task offloading and spectrum allocation in UAV collaborative edge computing

Abstract

Capable of flexibly supporting diverse applications and providing computation services, the Mobile Edge Computing (MEC)-assisted Unmanned Aerial Vehicle (UAV) network is emerging as an innovational paradigm. In this paradigm, the heterogeneous resources of the network, including computing and communication resources, should be allocated properly to reduce computation and communication latency as well as energy consumption. However, most existing works solely focus on the optimization issues with global information, which is generally difficult to obtain in real-world scenarios. In this paper, fully considering the incomplete information resulting from diverse types of tasks, we study the joint task offloading and spectrum allocation problem in UAV network, where free UAV nodes serve as helpers for cooperative computation. The objective is to jointly optimize offloading mode, collaboration pairing, and channel allocation to minimize the weighted network cost. To achieve the purpose with only partial observation, an extensive-form game is introduced to reformulate the problem, and a regret learning-based scheme is proposed to achieve the equilibrium solution. With retrospective improvement property and information set concept, the designed algorithm is capable of combating incomplete information and obtaining more precise allocation patterns for diverse tasks. Numerical results show that our proposed algorithm outperforms the benchmarks across various settings.

Keywords

UAV networks / Edge computing / Task offloading / Spectrum allocation / Partial observation / Regret learning

References

[1] M. Mozaffari, W. Saad, M. Bennis, Y.H. Nam, M. Debbah, A tutorial on UAVs for wireless networks: applications, challenges, and open problems, IEEE Commun. Surv. Tutor. 21 (3) (2019) 2334-2360.

[2] S. Zhang, Y. Zeng, R. Zhang, Cellular-enabled UAV communication: a connectivity-constrained trajectory optimization perspective, IEEE Trans. Commun. 67 (3) (2019) 2580-2604.

[3] W. Liu, B. Cao, M. Peng, Blockchain based offloading strategy: incentive, effective-ness and security, IEEE J. Sel. Areas Commun. 40 (12) (2022) 3533-3546.

[4] M. Cao, L. Zhang, B. Cao, Toward on-device federated learning: a direct acyclic graph-based blockchain approach, IEEE Trans. Neural Netw. Learn. Syst. 34 (4)(2023) 2028-2042.

[5] B. Cao, Z. Wang, L. Zhang, D. Feng, M. Peng, L. Zhang, Z. Han, Blockchain systems, technologies, and applications: a methodology perspective, IEEE Commun. Surv. Tutor. 25 (1) (2023) 353-385.

[6] X. Tang, C. Cao, Y. Wang, S. Zhang, Y. Liu, M. Li, T. He, Computing power net-work: the architecture of convergence of computing and networking towards 6G requirement, China Commun. 18 (2) (2021) 175-185.

[7] D. Sabella, A. Vaillant, P. Kuure, U. Rauschenbach, F. Giust, Mobile-edge computing architecture: the role of MEC in the Internet of things, IEEE Consum. Electron. Mag. 5(4) (2016) 84-91.

[8] S. He, X. Lyu, W. Ni, H. Tian, R.P. Liu, E. Hossain, Virtual service placement for edge computing under finite memory and bandwidth, IEEE Trans. Commun. 68 (12)(2020) 7702-7718.

[9] Y. Ren, Y. Sun, M. Peng, Deep reinforcement learning based computation offloading in fog enabled industrial internet of things, IEEE Trans. Ind. Inform. 17 (7) (2021) 4978-4987.

[10] G. Faraci, C. Grasso, G. Schembra, Design of a 5G network slice extension with MEC UAVs managed with reinforcement learning, IEEE J. Sel. Areas Commun. 38 (10)(2020) 2356-2371.

[11] F. Jiang, K. Wang, L. Dong, C. Pan, W. Xu, K. Yang, AI driven heterogeneous MEC system with UAV assistance for dynamic environment: challenges and solutions, IEEE Netw. 35 (1) (2020) 400-408.

[12] X. Huang, X. Yang, Q. Chen, J. Zhang, Task offloading optimization for UAV-assisted fog-enabled internet of things networks, IEEE Int. Things J. 9(2) (2021) 1082-1094.

[13] Z. Yu, Y. Gong, S. Gong, Y. Guo, Joint task offloading and resource allocation in UAV-enabled mobile edge computing, IEEE Int. Things J. 7(4) (2020) 3147-3159.

[14] Z. Cheng, Z. Gao, M. Liwang, L. Huang, X. Du, M. Guizani, Intelligent task offloading and energy allocation in the UAV-aided mobile edge-cloud continuum, IEEE Netw. 35 (5) (2021) 42-49.

[15] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, L. Hanzo, Multi-agent deep rein-forcement learning-based trajectory planning for multi-UAV assisted mobile edge computing, IEEE Trans. Cogn. Commun. Netw. 7(1) (2021) 73-84.

[16] N. Zhao, Z. Ye, Y. Pei, Y.-C. Liang, D. Niyato, Multi-agent deep reinforcement learn-ing for task offloading in UAV-assisted mobile edge computing, IEEE Trans. Wirel. Commun. 21 (9) (2022) 6949-6960.

[17] A. Sacco, F. Esposito, G. Marchetto, P. Montuschi, Sustainable task offloading in UAV networks via multi-agent reinforcement learning, IEEE Trans. Veh. Technol. 70 (5) (2021) 5003-5015.

[18] Z. Ning, P. Dong, M. Wen, X. Wang, L. Guo, R.Y. Kwok, H.V. Poor, 5G-enabled UAV-to-community offloading: joint trajectory design and task scheduling, IEEE J. Sel. Areas Commun. 39 (11) (2021) 3306-3320.

[19] W. Liu, B. Li, W. Xie, Y. Dai, Z. Fei, Energy efficient computation offloading in aerial edge networks with multi-agent cooperation, IEEE Trans. Wirel. Commun. 22 (9)(2023) 5725-5739.

[20] M.A. Hossain, A.R. Hossain, N. Ansari, Numerology-capable UAV-MEC for future generation massive IoT networks, IEEE Int. Things J. 9 (23) (2022) 23860-23868.

[21] Q. Qi, X. Chen, C. Zhong, Z. Zhang, Integration of energy, computation and com-munication in 6G cellular internet of things, IEEE Commun. Lett. 24 (6) (2020) 1333-1337.

[22] C. Fan, C. She, H. Zhang, B. Li, C. Zhao, D. Niyato, Learning to optimize user asso-ciation and spectrum allocation with partial observation in mmwave-enabled UAV networks, IEEE Trans. Wirel. Commun. 21 (8) (2022) 5873-5888.

[23] M.M. Azari, G. Geraci, A. Garcia-Rodriguez, S. Pollin, UAV-to-UAV communications in cellular networks, IEEE Trans. Wirel. Commun. 19 (9) (2020) 6130-6144.

[24] T.W. Neller, M. Lanctot, An introduction to counterfactual regret minimization, in: Proceedings of Model AI Assignments, the Fourth Symposium on Educational Ad-vances in Artificial Intelligence, EAAI, 2013, pp. 1-38.

[25] T. Fang, F. Yuan, L. Ao, J. Chen, Joint task offloading, D2D pairing, and resource allocation in device-enhanced MEC: a potential game approach, IEEE Int. Things J. 9(5) (2021) 3226-3237.

[26] X. Lin, J.G. Andrews, A. Ghosh, Spectrum sharing for device-to-device communica-tion in cellular networks, IEEE Trans. Wirel. Commun. 13 (12) (2014) 6727-6740.

[27] X. Lyu, H. Tian, W. Ni, Y. Zhang, P. Zhang, R.P. Liu, Energy-efficient admission of delay-sensitive tasks for mobile edge computing, IEEE Trans. Commun. 66 (6)(2018) 2603-2616.

[28] M. Qin, N. Cheng, Z. Jing, T. Yang, W. Xu, Q. Yang, R.R. Rao, Service-oriented energy-latency tradeoff for IoT task partial offloading in MEC-enhanced multi-RAT networks, IEEE Int. Things J. 8(3) (2020) 1896-1907.

[29] Y. Sun, S. Chen, Z. Wang, S. Mao, A joint learning and game-theoretic approach to multi-dimensional resource management in fog radio access networks, IEEE Trans. Veh. Technol. 72 (2) (2023) 2550-2563.

[30] Y. Bai, C. Jin, S. Mei, T. Yu, Near-optimal learning of extensive-form games with imperfect information, in: International Conference on Machine Learning, PMLR, 2022, pp. 1337-1382.

[31] V. Hakami, M. Dehghan, Learning stationary correlated equilibria in constrained general-sum stochastic games, IEEE Trans. Cybern. 46 (7) (2016) 1640-1654.

[32] Q. Liu, T. Yu, Y. Bai, C. Jin, A sharp analysis of model-based reinforcement learn-ing with self-play, in: International Conference on Machine Learning, PMLR, 2021, pp. 7001-7010.

[33] C. Fan, B. Li, C. Zhao, Y.-C. Liang, Regret matching learning based spectrum reuse in small cell networks, IEEE Trans. Veh. Technol. 69 (1) (2020) 1060-1064.

[34] S. Srinivasan, M. Lanctot, V. Zambaldi, J. Perolat, K. Tuyls, R. Munos, M. Bowling, Actor-critic policy optimization in partially observable multiagent environments, in:Neural Information Processing Systems (NIPS), Curran Associates, Inc., 2018, pp. 1-14.

[35] N. Brown, T. Sandholm, Solving imperfect-information games via discounted re-gret minimization, in: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, 2019, pp. 1829-1836.

[36] Y. He, J. Ren, G. Yu, Y. Cai, D2D communications meet mobile edge computing for enhanced computation capacity in cellular networks, IEEE Trans. Wirel. Commun. 18 (3) (2019) 1750-1763.