A pipelining task offloading strategy via delay-aware multi-agent reinforcement learning in Cybertwin-enabled 6G network

Abstract

Cybertwin-enabled 6th Generation (6G) network is envisioned to support artificial intelligence-native management to meet changing demands of 6G applications. Multi-Agent Deep Reinforcement Learning (MADRL) technologies driven by Cybertwins have been proposed for adaptive task offloading strategies. However, the existence of random transmission delay between Cybertwin-driven agents and underlying networks is not considered in related works, which destroys the standard Markov property and increases the decision reaction time to reduce the task offloading strategy performance. In order to address this problem, we propose a pipelining task offloading method to lower the decision reaction time and model it as a delay-aware Markov Decision Process (MDP). Then, we design a delay-aware MADRL algorithm to minimize the weighted sum of task execution latency and energy consumption. Firstly, the state space is augmented using the lastly-received state and historical actions to rebuild the Markov property. Secondly, Gate Transformer-XL is introduced to capture historical actions' importance and maintain the consistent input dimension dynamically changed due to random transmission delays. Thirdly, a sampling method and a new loss function with the difference between the current and target state value and the difference between real state-action value and augmented state-action value are designed to obtain state transition trajectories close to the real ones. Numerical results demonstrate that the proposed methods are effective in reducing reaction time and improving the task offloading performance in the random-delay Cybertwin-enabled 6G networks.

Keywords

Cybertwin / Multi-Agent Deep Reinforcement Learning (MADRL) / Task offloading / Pipelining / Delay-aware

Declaration of Competing Interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled “A pipelining task offloading strategy via delay-aware multi-agent reinforcement learning in Cybertwin-enabled 6G network”.

Acknowledgement

This research was funded by the National Key Research and Development Program of China under Grant 2019YFB1803301 and Beijing Natural Science Foundation (L202002).

References

[1] E. Sisinni, A. Saifullah, S. Han, U. Jennehag, M. Gidlund, Industrial Internet of Things: challenges, opportunities, and directions, IEEE Trans. Ind. Inform. 14 (11) (2018) 4724-4734.

[2] Z. Cheng, M. Min, M. Liwang, L. Huang, Z. Gao, Multiagent ddpg-based joint task partitioning and power control in fog computing networks, IEEE Int. Things J. 9 (1) (2021) 104-116.

[3] T.X. Tran, A. Hajisami, P. Pandey, D. Pompili, Collaborative mobile edge computing in 5G networks: new paradigms, scenarios, and challenges, IEEE Commun. Mag. 55 (4) (2017) 54-61.

[4] H. Liao, Z. Jia, Z. Zhou, Y. Wang, H. Zhang, S. Mumtaz, Cloud-edge-end collabora-tion in air-ground integrated power iot: a semidistributed learning approach, IEEE Trans. Ind. Inform. 18 (11) (2022) 8047-8057.

[5] A. Hazra, P.K. Donta, T. Amgoth, S. Dustdar, Cooperative transmission scheduling and computation offloading with collaboration of fog and cloud for industrial iot applications, IEEE Int. Things J. 10 (5) (2023) 3944-3953.

[6] F. Zhang, G. Han, L. Liu, M. Martinez-Garcia, Y. Peng, Deep reinforcement learning based cooperative partial task offloading and resource allocation for iiot applica-tions, IEEE Trans. Netw. Sci. Eng. 10 (5) (2023) 2991-3006.

[7] W. Wei, H. Gu, K. Wang, J. Li, X. Zhang, N. Wang, Multi-dimensional resource allocation in distributed data centers using deep reinforcement learning, IEEE Trans. Netw. Serv. Manag. 20 (2) (2023) 1817-1829.

[8] N.C. Luong, D.T. Hoang, S. Gong, D. Niyato, P. Wang, Y.-C. Liang, D.I. Kim, Applica-tions of deep reinforcement learning in communications and networking: a survey, IEEE Commun. Surv. Tutor. 21 (4) (2019) 3133-3174.

[9] Y. Mao, C. You, J. Zhang, K. Huang, K.B. Letaief, A survey on mobile edge com-puting: the communication perspective, IEEE Commun. Surv. Tutor. 19 (4) (2017) 2322-2358.

[10] Z. Ning, P. Dong, X. Kong, F. Xia, A cooperative partial computation offloading scheme for mobile edge computing enabled internet of things, IEEE Int. Things J. 6 (3) (2018) 4804-4814.

[11] Q. Yu, J. Ren, Y. Fu, Y. Li, W. Zhang, Cybertwin: an origin of next generation net-work architecture, IEEE Trans. Wirel. Commun. 26 (6) (2019) 111-117.

[12] Q. Yu, J. Ren, H. Zhou, W. Zhang, A cybertwin based network architecture for 6G, in: Proceedings of the 2020 2nd 6G Wireless Summit, 6G SUMMIT, IEEE, 2020, pp. 1-5.

[13] S. Nath, M. Baranwal, H. Khadilkar, Revisiting state augmentation methods for reinforcement learning with stochastic delays, in: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, ACM, 2021, pp. 1346-1355.

[14] M. Agarwal, V. Aggarwal, Blind decision making: reinforcement learning with de-layed observations, Pattern Recogn. Lett. 150 (2021) 176-182.

[15] P. Liotet, E. Venneri, M. Restelli, Learning a belief representation for delayed rein-forcement learning, in: Proceedings of the 2021 International Joint Conference on Neural Networks, IJCNN, IEEE, 2021, pp. 1-8.

[16] Y. Bouteiller, S. Ramstedt, G. Beltrame, C. Pal, J. Binas, Reinforcement learning with random delays, in: Proceedings of the 2021 International Conference on Learning Representations, OpenReview.net, 2021.

[17] B. Chen, M. Xu, L. Li, D. Zhao, Delay-aware model-based reinforcement learning for continuous control, Neurocomputing 450 (2021) 119-128.

[18] L. Huang, S. Bi, Y.-J.A. Zhang, Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks, IEEE Trans. Mob. Comput. 19 (11) (2019) 2581-2593.

[19] J. Zhang, X. Hu, Z. Ning, E.C.-H. Ngai, L. Zhou, J. Wei, J. Cheng, B. Hu, V.C. Leung, Joint resource allocation for latency-sensitive services over mobile edge computing networks with caching, IEEE Int. Things J. 6 (3) (2018) 4283-4294.

[20] K. Wang, W. Chen, J. Li, Y. Yang, L. Hanzo, Joint task offloading and caching for massive mimo-aided multi-tier computing networks, IEEE Trans. Commun. 70 (3) (2022) 1820-1833.

[21] C. Ding, J.-B. Wang, H. Zhang, M. Lin, G.Y. Li, Joint optimization of transmission and computation resources for satellite and high altitude platform assisted edge computing, IEEE Trans. Wirel. Commun. 21 (2) (2021) 1362-1377.

[22] M. Chen, Y. Hao, Task offloading for mobile edge computing in software defined ultra-dense network, IEEE J. Sel. Areas Commun. 36 (3) (2018) 587-597.

[23] J. Su, S. Yu, B. Li, Y. Ye, Distributed and collective intelligence for computation offloading in aerial edge networks, IEEE Trans. Intell. Transp. Syst. 24 (7) (2023) 7516-7526.

[24] T. Alfakih, M.M. Hassan, A. Gumaei, C. Savaglio, G. Fortino, Task offloading and re-source allocation for mobile edge computing by deep reinforcement learning based on sarsa, IEEE Access 8 (2020) 54074-54084, https://doi.org/10.1109/ACCESS.2020.2981434.

[25] M. Tang, V.W. Wong, Deep reinforcement learning for task offloading in mobile edge computing systems, IEEE Trans. Mob. Comput. 21 (6) (2020) 1985-1997.

[26] J. Yan, S. Bi, Y.J.A. Zhang, Offloading and resource allocation with general task graph in mobile edge computing: a deep reinforcement learning approach, IEEE Trans. Wirel. Commun. 19 (8) (2020) 5404-5419.

[27] B. Li, Y. Liu, L. Tan, H. Pan, Y. Zhang, Digital twin assisted task offloading for aerial edge computing and networks, IEEE Trans. Veh. Technol. 71 (10) (2022) 10863-10877.

[28] X. Zhong, Y. He, A cybertwin-driven task offloading scheme based on deep rein-forcement learning and graph attention networks, in: Proceedings of the 2021 13th International Conference on Wireless Communications and Signal Processing, WCSP, IEEE, 2021, pp. 1-6.

[29] M. Adhikari, A. Munusamy, N. Kumar, S.N. Srirama, Cybertwin-driven resource pro-visioning for ioe applications at 6G-enabled edge networks, IEEE Trans. Ind. Inform. 18 (7) (2021) 4850-4858.

[30] W. Hou, H. Wen, H. Song, W. Lei, W. Zhang, Multiagent deep reinforcement learning for task offloading and resource allocation in cybertwin-based networks, IEEE Int. Things J. 8 (22) (2021) 16256-16268.

[31] T.K. Rodrigues, J. Liu, N. Kato, Application of cybertwin for offloading in mobile multiaccess edge computing for 6G networks, IEEE Int. Things J. 8 (22) (2021) 16231-16242.

[32] H. Peng, X. Shen, Multi-agent reinforcement learning based resource management in mec- and uav-assisted vehicular networks, IEEE J. Sel. Areas Commun. 39 (1) (2020) 131-141.

[33] N. Zhao, Z. Ye, Y. Pei, Y.-C. Liang, D. Niyato, Multi-agent deep reinforcement learn-ing for task offloading in uav-assisted mobile edge computing, IEEE Trans. Wirel. Commun. 21 (9) (2022) 6949-6960.

[34] E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R.L. Kaufman, A. Clark, S. Noury, et al., Stabilizing transformers for reinforcement

[35] learning, in: Proceedings of the 2020 International Conference on Machine Learning, PMLR, 2020, pp. 7487-7498.

[36] S. Fujimoto, H. Hoof, D. Meger, Addressing function approximation error in actor-critic methods, in: Proceedings of the 2018 International Conference on Machine Learning, PMLR, 2018, pp. 1587-1596.

[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 31st Annual Conference on Neural Information Processing Systems, NIPS, 2017, pp. 5998-6008.

[38] R. Lowe, Y. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, I. Mordatch, Multi-agent actor-critic for mixed cooperative-competitive environments, in: Proceedings of the 31st Annual Conference on Neural Information Processing Systems, NIPS, 2017, pp. 6379-6390.