Cross-domain resources optimization for hybrid edge computing networks: Federated DRL approach

Abstract

Due to the dynamic nature of service requests and the uneven distribution of services in the Internet of Vehicles (IoV), Multi-access Edge Computing (MEC) networks with pre-installed servers are often susceptible to insufficient computing power at certain times or in certain areas. In addition, Vehicular Users (VUs) need to share their observations for centralized neural network training, resulting in additional communication overhead. In this paper, we present a hybrid MEC server architecture, where fixed RoadSide Units (RSUs) and Mobile Edge Servers (MESs) cooperate to provide computation offloading services to VUs. We propose a distributed federated learning and Deep Reinforcement Learning (DRL) based algorithm, namely Federated Dueling Double Deep Q-Network (FD3QN), with the objective of minimizing the weighted sum of service latency and energy consumption. Horizontal federated learning is incorporated into the Dueling Double Deep Q-Network (D3QN) to allocate cross-domain resources after the offload decision process. A client-server framework with federated aggregation is used to maintain the global model. The proposed FD3QN algorithm can jointly optimize power, sub-band, and computational resources. Simulation results show that the proposed algorithm outperforms baselines in terms of system cost and exhibits better robustness in uncertain IoV environments.

Keywords

Internet of vehicles / Multi-access edge computing / Cross-domain resources optimization / Federated learning / Dueling double deep Q-network

References

[1] S. Raza, S. Wang, M. Ahmed, M.R. Anwar, M.A. Mirza, W.U. Khan, Task offloading and resource allocation for IoV using 5G NR V2X communication, IEEE Int. Things J. 9 (13) (2022) 10397-10410.

[2] Y. Gong, Y. Wei, Z. Feng, F.R. Yu, Y. Zhang, Resource allocation for integrated sensing and communication in digital twin enabled Internet of vehicles, IEEE Trans. Veh. Technol. 72 (4) (2023) 4510-4524.

[3] X. Zhang, W. Wu, Z. Zhao, J. Wang, S. Liu, RMDDQN-learning: computation offload-ing algorithm based on dynamic adaptive multi-objective reinforcement learning in Internet of vehicles, IEEE Trans. Veh. Technol. 72 (9) (2023) 11374-11388.

[4] B. Hazarika, K. Singh, C.-P. Li, S. Biswas, Multi-agent DRL-based computation of-floading in multiple RIS-aided IoV networks, in: 2022 IEEE Military Communica-tions Conference (MILCOM), IEEE, 2022, pp. 1-6.

[5] X. Ye, M. Li, P. Si, R. Yang, Z. Wang, Y. Zhang, Collaborative and intelligent resource optimization for computing and caching in IoV with blockchain and MEC using A3C approach, IEEE Trans. Veh. Technol. 72 (2) (2023) 1449-1463.

[6] Shichao Xia, Zhixiu Yao, Yun Li, Shiwen Mao, Online distributed offloading and computing resource management with energy harvesting for heterogeneous MEC-enabled IoT, IEEE Trans. Wirel. Commun. 10 (20) (2021) 6743-6757.

[7] Y. Li, H. Ma, L. Wang, S. Mao, G. Wang, Optimized content caching and user associ-ation for edge computing in densely deployed heterogeneous networks, IEEE Trans. Mob. Comput. 21 (6) (2022) 2130-2142.

[8] X. Yuan, J. Chen, N. Zhang, J. Ni, F.R. Yu, V.C.M. Leung, Digital twin-driven vehic-ular task offloading and IRS configuration in the Internet of vehicles, IEEE Trans. Intell. Transp. Syst. 23 (12) (2022) 24290-24304.

[9] M. Guo, W. Wang, X. Huang, Y. Chen, L. Zhang, L. Chen, Lyapunov-based partial computation offloading for multiple mobile devices enabled by harvested energy in MEC, IEEE Int. Things J. 9 (11) (2022) 9025-9035.

[10] Y. Hou, C. Wang, M. Zhu, X. Xu, X. Tao, X. Wu, Joint allocation of wireless re-source and computing capability in MEC-enabled vehicular network, China Com-mun. 18 (6) (2021) 64-76.

[11] Y. Li, S. Xia, M. Zheng, B. Cao, Q. Liu, Lyapunov optimization-based trade-off policy for mobile cloud offloading in heterogeneous wireless networks, IEEE Trans. Cloud Comput. 10 (1) (2022) 491-505.

[12] X. Deng, J. Yin, P. Guan, N.N. Xiong, L. Zhang, S. Mumtaz, Intelligent delay-aware partial computing task offloading for multiuser industrial Internet of things through edge computing, IEEE Int. Things J. 10 (4) (2023) 2954-2966.

[13] H. Hu, D. Wu, F. Zhou, X. Zhu, R.Q. Hu, H. Zhu, Intelligent resource allocation for edge-cloud collaborative networks: a hybrid DDPG-D3QN approach, IEEE Trans. Veh. Technol. 72 (8) (2023) 10696-10709.

[14] S.S. Shinde, D. Tarchi, Collaborative reinforcement learning for multi-service Inter-net of vehicles, IEEE Int. Things J. 10 (3) (2023) 2589-2602.

[15] H. Xiao, W. Zhang, W. Li, A.T. Chronopoulos, Z. Zhang, Joint clustering and blockchain for real-time information security transmission at the crossroads in C-V2X networks, IEEE Int. Things J. 8 (18) (2021) 13926-13938.

[16] H. Guo, L. Rui, Z. Gao, V2V task offloading algorithm with LSTM-based spatiotempo-ral trajectory prediction model in SVCNs, IEEE Trans. Veh. Technol. 71 (10) (2022) 11017-11032.

[17] X. Dai, X. Zhu, H. Jiang, A learning-based approach for vehicle-to-vehicle computa-tion offloading, IEEE Int. Things J. 10 (8) (2023) 7244-7258.

[18] Y. Ming, J. Chen, Y. Dong, Z. Wang, Evolutionary game based strategy selection for hybrid V2V communications, IEEE Trans. Veh. Technol. 71 (2) (2022) 2128-2133.

[19] H. Wang, T. Lv, Z. Lin, J. Zeng, Energy-delay minimization of task migration based on game theory in MEC-assisted vehicular networks, IEEE Trans. Veh. Technol. 71 (8) (2022) 8175-8188.

[20] S. Pang, N. Wang, M. Wang, S. Qiao, X. Zhai, N.N. Xiong, A smart network resource management system for high mobility edge computing in 5G Internet of vehicles, IEEE Trans. Netw. Sci. Eng. 8(4) (2021) 3179-3191.

[21] H. Zhang, Z. Wang, K. Liu, V2X offloading and resource allocation in SDN-assisted MEC-based vehicular networks, China Commun. 17 (5) (2020) 266-283.

[22] Z. Qin, G.Y. Li, H. Ye, Federated learning and wireless communications, IEEE Wirel. Commun. 28 (5) (2021) 134-140.

[23] J. Posner, L. Tseng, M. Aloqaily, Y. Jararweh, Federated learning in vehicular net-works: opportunities and solutions, IEEE Netw. 35 (2) (2021) 152-159.

[24] B. Ghimire, D.B. Rawat, Recent advances on federated learning for cybersecurity and cybersecurity for federated learning for Internet of things, IEEE Int. Things J. 9 (11) (2022) 8229-8249.

[25] Q. Zhang, H. Wen, Y. Liu, S. Chang, Z. Han, Federated-reinforcement-learning-enabled joint communication, sensing, and computing resources allocation in connected automated vehicles networks, IEEE Int. Things J. 9 (22) (2022) 23224-23240.

[26] Y. Lee, S. Park, J.-H. Ahn, J. Kang, Accelerated federated learning via greedy aggre-gation, IEEE Commun. Lett. 26 (12) (2022) 2919-2923.

[27] W. Feng, S. Lin, N. Zhang, G. Wang, B. Ai, L. Cai, Joint C-V2X based offloading and resource allocation in multi-tier vehicular edge computing system, IEEE J. Sel. Areas Commun. 41 (2) (2023) 432-445.

[28] S. Narayanan, O. Liberg, A. Höglund, D. Tsolkas, N. Passas, L. Merakos, Sidelink optimizations for layer-3-based IoT relaying in 5G NR, IEEE Int. Things J. 5(2)(2022) 140-145.

[29] X. Dai, Z. Xiao, H. Jiang, A learning-based approach for vehicle-to-vehicle compu-tation offloading, IEEE Int. Things J. 10 (8) (2023) 7244-7258.

[30] Shumo Wang, Xiaoqin Song, Han Xu, Tiecheng Song, Guowei Zhang, Yang Yang, Joint offloading decision and resource allocation in vehicular edge computing net-works, Digital Commun. Net. 11 (1) (2025) 71-82.

[31] Q. Luo, C. Li, T.H. Luan, W. Shi, W. Wu, Self-learning based computation offloading for Internet of vehicles: model and algorithm, IEEE Trans. Wirel. Commun. 20 (9)(2021) 5913-5925.

[32] W. Huang, Z. Zeng, N. Xiong, S. Mumtaz, JOET: sustainable vehicle-assisted edge computing for IoT devices, J. Syst. Archit. 131 (2022).

[33] Abuzar B.M. Adam, Xiaoyu Wan, Zhengqiang Wang, User scheduling and power allocation for downlink multi-cell multi-carrier NOMA systems, Digital Commun. Net. 9(1) (2023) 252-263.

[34] Z. Wu, G. Ishigaki, R. Gour, C. Li, J.P. Jue, Reinforcement learning-based network slice resource allocation for federated learning applications, in: 2022 IEEE Global Communications Conference (GLOBECOM), IEEE, 2022, pp. 3647-3652.

[35] H. Sharma, N. Kumar, R. Tekchandani, Mitigating jamming attack in 5G hetero-geneous networks: a federated deep reinforcement learning approach, IEEE Trans. Veh. Technol. 72 (2) (2023) 2439-2452.

[36] Y. Ji, Y. Wang, H. Zhao, G. Gui, H. Gacanin, Multi-agent reinforcement learning resources allocation method using dueling double deep Q-network in vehicular net-works, IEEE Trans. Veh. Technol. 72 (10) (2023) 13447-13460.

[37] S. Zarandi, H. Tabassum, Federated double deep Q-learning for joint delay and energy minimization in IoT networks, in: 2021 IEEE International Conference on Communications Workshops (ICC Workshops), IEEE, 2021, pp. 1-6.

[38] M.Z. Alam, A. Jamalipour, Multi-agent DRL-based Hungarian algorithm (MADRLHA) for task offloading in multi-access edge computing Internet of vehi-cles (IoVs), IEEE Trans. Wirel. Commun. 21 (9) (2022) 7641-7652.