Joint offloading decision and resource allocation in vehicular edge computing networks

Abstract

With the rapid development of Intelligent Transportation Systems (ITS), many new applications for Intelligent Connected Vehicles (ICVs) have sprung up. In order to tackle the conflict between delay-sensitive applications and resource-constrained vehicles, computation offloading paradigm that transfers computation tasks from ICVs to edge computing nodes has received extensive attention. However, the dynamic network conditions caused by the mobility of vehicles and the unbalanced computing load of edge nodes make ITS face challenges. In this paper, we propose a heterogeneous Vehicular Edge Computing (VEC) architecture with Task Vehicles (TaVs), Service Vehicles (SeVs) and Roadside Units (RSUs), and propose a distributed algorithm, namely PG-MRL, which jointly optimizes offloading decision and resource allocation. In the first stage, the offloading decisions of TaVs are obtained through a potential game. In the second stage, a multi-agent Deep Deterministic Policy Gradient (DDPG), one of deep reinforcement learning algorithms, with centralized training and distributed execution is proposed to optimize the real-time transmission power and subchannel selection. The simulation results show that the proposed PG-MRL algorithm has significant improvements over baseline algorithms in terms of system delay.

Keywords

Computation offloading / Resource allocation / Vehicular edge computing / Potential game / Multi-agent deep deterministic policy gradient

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This project is supported by Future Network Scientific Research Fund Project (FNSRFP-2021-ZD-4), National Natural Science Foundation of China (No. 61991404, 61902182), National Key Research and Development Program of China under Grant 2020YFB1600104, Key Research and Development Plan of Jiangsu Province under Grant BE2020084-2.

References

[1] S. Gurugopinath, P.C. Sofotasios, Y. Al-Hammadi, S. Muhaidat, Cache-aided nonorthogonal multiple access for 5G-enabled vehicular networks, IEEE Trans. Veh. Technol. 68 (9) (2019) 8359-8371.

[2] L. Liang, H. Ye, G.Y. Li, Spectrum sharing in vehicular networks based on multiagent reinforcement learning, IEEE J. Sel. Area. Commun. 37 (10) (2019) 2282-2292.

[3] D.C. Nguyen, M. Ding, P.N. Pathirana, et al., 6G internet of things: a comprehensive survey, IEEE Internet Things J. 9 (1) (2022) 359-383.

[4] B. Lin, K. Lin, C. Lin, L. Yu, et al., Computation offloading strategy based on deep reinforcement learning for connected and autonomous vehicle in vehicular edge computing, J. Cloud Comput. 10 (1) (2021) 33.

[5] J.S. Choo, M. Kim, S. Pack, G. Dan, The software-defined vehicular cloud: a new level of sharing the road, IEEE Veh. Technol. Mag. 12 (2) (2017) 78-88.

[6] E. Peltonen, M. Bennis, M. Capobianco, M. Debbah, et al., 6G White Paper on Edge Intelligence, (White paper). https://doi.org/10.48550/arXiv.2004.14850. (Accessed 13 June 2022).

[7] Y. Xiao, G. Shi, Y. Li, W. Saad, H.V. Poor, Toward self learning edge intelligence in 6G, IEEE Commun. Mag. 58 (12) (2020) 34-40.

[8] N. Abbas, Y. Zhang, A. Taherkordi, T. Skeie, Mobile edge computing: a survey, IEEE Internet Things J. 5 (1) (2018) 450-465.

[9] S. Bitam, A. Mellouk, S. Zeadally, VANET-cloud: a generic cloud computing model for vehicular ad hoc networks, IEEE Wireless Commun. 22 (1) (2015) 96-102.

[10] Q. Wu, S. Nie, P. Fan, H. Liu, F. Qiang, Z. Li, A swarming approach to optimize the one-hop delay in smart driving inter-platoon communications, Sensors 18 (10) (2018) 3307.

[11] X. Gao, X. Huang, S. Bian, Z. Shao, Y. Yang, PORA: predictive offloading and resource allocation in dynamic fog computing systems, IEEE Internet Things J. 7 (1) (2020) 72-87.

[12] K. Wang, Y. Zhou, Z. Liu, Z. Shao, X. Luo, Y. Yang, Online task scheduling and resource allocation for intelligent NOMA-based industrial internet of things, IEEE J. Sel. Area. Commun. 38 (5) (2020) 803-815.

[13] K. Zheng, H. Meng, P. Chatzimisios, L. Lei, X. Shen, An SMDP-based resource allocation in vehicular cloud computing systems, IEEE Trans. Ind. Electron. 62 (12) (2015) 7920-7928.

[14] M.S. Bute, P. Fan, L. Zhang, F. Abbas, An efficient distributed task offloading scheme for vehicular edge computing networks, IEEE Trans. Veh. Technol. 70 (12) (2021) 13149-13161.

[15] Y. Liu, S. Wang, Q. Zhao, S. Du, et al., Dependency-aware task scheduling in vehicular edge computing, IEEE Internet Things J. 7 (6) (2020) 4961-4971.

[16] Y. Jang, J. Na, S. Jeong, J. Kang,Energy-efficient task offloading for vehicular edge computing: joint optimization of offloading and bit allocation, in: 2020 IEEE 91^st Vehicular Technology Conference (VTC2020-Spring), IEEE, 2020, pp. 1-5.

[17] Y. Wang, P. Lang, D. Tian, J. Zhou, et al., A game-based computation offloading method in vehicular multiaccess edge computing networks, IEEE Internet Things J. 7 (6) (2020) 4987-4996.

[18] G. Qiao, S. Leng, K. Zhang, Y. He, Collaborative task offloading in vehicular edge multi-access networks, IEEE Commun. Mag. 56 (8) (2018) 48-54.

[19] W. Fan, J. Liu, M. Hua, F. Wu, Y. Liu, Joint task offloading and resource allocation for multi-access edge computing assisted by parked and moving vehicles, IEEE Trans. Veh. Technol. 71 (5) (2022) 5314-5330.

[20] Q. Wu, H. Liu, R. Wang, P. Fan, Q. Fan, Z. Li, Delay-sensitive task offloading in the 802.11p-based vehicular fog computing systems, IEEE Internet Things J. 7 (1) (2020) 773-785.

[21] F. Sun, F. Hou, N. Cheng, M. Wang, H. Zhou, L. Gui, X. Shen, Cooperative task scheduling for computation offloading in vehicular cloud, IEEE Trans. Veh. Technol. 67 (11) (2018) 11049-11061.

[22] Z. Yu, Y. Tang, L. Zhang, H. Zeng, Deep reinforcement learning based computing offloading decision and task scheduling in internet of vehicles, in: 2021 IEEE/CIC International Conference on Communications in China (ICCC), IEEE, 2021, pp. 1166-1171.

[23] M.Z. Alam, A. Jamalipour, Multi-agent DRL-based Hungarian algorithm (MADRLHA) for task offloading in multi-access edge computing internet of vehicles (IoVs), IEEE Trans. Wireless Commun. 21 (9) (2022) 7641-7652.

[24] J. Zhao, Q. Li, Y. Gong, K. Zhang, Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks, IEEE Trans. Veh. Technol. 68 (8) (2019) 7944-7956.

[25] C. Chen, H. Li, H. Li, R. Fu, Y. Liu, S. Wan, Efficiency and fairness oriented dynamic task offloading in internet of vehicles, IEEE Trans. Green Commun. Netw. 6 (3) (2022) 1481-1493.

[26] Y. Liu, H. Yu, S. Xie, Y. Zhang, Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks, IEEE Trans. Veh. Technol. 68 (11) (2019) 11158-11168.

[27] S. Yu, X. Chen, Z. Zhou, X. Gong, D. Wu, When deep reinforcement learning meets federated learning: intelligent multitimescale resource management for multi-access edge computing in 5g ultradense network, IEEE Internet Things J. 8 (4) (2021) 2238-2251.

[28] W.-J. Wang, H.-C. Yang, M.-S. Alouini, Wireless transmission of big data: a transmission time analysis over fading channel, IEEE Trans. Wireless Commun. 17 (7) (2018) 4315-4325.

[29] D. Monderer, L.S. Shapley, Potential games, Game. Econ. Behav. 14 (1) (1996) 124-143.

[30] Z. Han, D. Niyato, W. Saad, T. Basar, A. Hjorungnes, Game Theory in Wireless and Communication Networks: Theory, Models, and Applications, Cambridge Univ. Press, Cambridge, U.K., 2012.

[31] P. Li, H.B. Duan, A potential game approach to multiple UAV cooperative search and surveillance, Aerosp.Sci. Technol. 68 (2017) 403-415.

[32] Y. Yang, Z. Liu, X. Yang, K. Wang, X. Hong, X. Ge, POMT: paired offloading of multiple tasks in heterogeneous fog networks, IEEE Internet Things J. 6 (5) (2019) 8658-8669.

[33] W. Hou, H. Wen, H. Song, W. Lei, W. Zhang, Multiagent deep reinforcement learning for task offloading and resource allocation in cybertwin-based networks, IEEE Internet Things J. 8 (22) (2021) 16256-16268.

[34] 3GPP TR 36.885 V14.0.0, Study LTE-Based V2X Services, http://www.3gpp.org, 2016 (accessed 13 Jun. 2022).

[35] Z. Liu, Y. Yang, M.-T. Zhou, Z. Li, A unified cross-entropy based task scheduling algorithm for heterogeneous fog networks, in: Proceedings of the 1st ACM International Workshop on Smart Cities and Fog Computing, ACM, 2018, pp. 1-6.

[36] R.Y. Rubinstein, D.P. Kroese, The Cross Entropy Method:A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation, Springer, New York, 2004.

[37] T. Fang, F. Yuan, L. Ao, J. Chen, Joint task offloading, D2D pairing, and resource allocation in device-enhanced MEC: a potential game approach, IEEE Internet Things J. 9 (5) (2022) 3226-3237.

[38] H. Peng, X. Shen, DDPG-based resource management for MEC/UAV-assisted vehicular networks, in: 2020 IEEE 92nd Vehicular Technology Conference, IEEE, 2020, pp. 1-6.