A knowledge graph-based reinforcement learning approach for cooperative caching in MEC-enabled heterogeneous networks^✩

Abstract

Existing wireless networks are flooded with video data transmissions, and the demand for high-speed and low-latency video services continues to surge. This has brought with it challenges to networks in the form of congestion as well as the need for more resources and more dedicated caching schemes. Recently, Multi-access Edge Computing (MEC)-enabled heterogeneous networks, which leverage edge caches for proximity delivery, have emerged as a promising solution to all of these problems. Designing an effective edge caching scheme is critical to its success, however, in the face of limited resources. We propose a novel Knowledge Graph (KG)-based Dueling Deep Q-Network (KG-DDQN) for cooperative caching in MEC-enabled heterogeneous networks. The KG-DDQN scheme leverages a KG to uncover video relations, providing valuable insights into user preferences for the caching scheme. Specifically, the KG guides the selection of related videos as caching candidates (i.e., actions in the DDQN), thus providing a rich reference for implementing a personalized caching scheme while also improving the decision efficiency of the DDQN. Extensive simulation results validate the convergence effectiveness of the KG-DDQN, and it also outperforms baselines regarding cache hit rate and service delay.

Keywords

Multi-access edge computing / Cooperative caching / Resource allocation / Knowledge graph / Reinforcement learning

References

[1] Y. Xu, G. Gui, H. Gacanin, F. Adachi, A survey on resource allocation for 5G hetero-geneous networks: current research, future trends, and challenges, IEEE Commun. Surv. Tutor. 23 (2) (2021) 668-695.

[2] S.K. Sharma, X. Wang, Toward massive machine type communications in ultra-dense cellular IoT networks: current issues and machine learning-assisted solutions, IEEE Commun. Surv. Tutor. 22 (1) (2020) 426-471.

[3] F. Zeng, Q. Chen, L. Meng, J. Wu, Volunteer assisted collaborative offloading and resource allocation in vehicular edge computing, IEEE Trans. Intell. Transp. Syst. 22 (6) (2021) 3247-3257.

[4] B. Jedari, G. Premsankar, G. Illahi, M.D. Francesco, A. Mehrabi, A. Ylä-Jääski, Video caching, analytics, and delivery at the wireless edge: a survey and future directions, IEEE Commun. Surv. Tutor. 23 (1) (2021) 431-471.

[5] H. Xiao, C. Xu, Z. Feng, R. Ding, S. Yang, L. Zhong, J. Liang, G.-M. Muntean, A transcoding-enabled 360° VR video caching and delivery framework for edge-enhanced next-generation wireless networks, IEEE J. Sel. Areas Commun. 40 (5) (2022) 1615-1631.

[6] L. Li, D. Shi, R. Hou, R. Chen, B. Lin, M. Pan, Energy-efficient proactive caching for adaptive video streaming via data-driven optimization, IEEE Internet Things J. 7 (6) (2020) 5549-5561.

[7] J. Luo, F.R. Yu, Q. Chen, L. Tang, Adaptive video streaming with edge caching and video transcoding over software-defined mobile networks: a deep reinforcement learning approach, IEEE Trans. Wirel.commun. 19 (3) (2020) 1577-1592.

[8] F. Li, Y. Lin, N. Peng, Y. Zhang,Deep reinforcement learning based computing of-floading for MEC-assisted heterogeneous vehicular networks, in:Proc. Int. Conf.commun. Technol., 2020, pp. 927-932.

[9] S. Wang, S. Bi, Y.-J.A. Zhang, Deep reinforcement learning with communication transformer for adaptive live streaming in wireless edge networks, IEEE J. Sel. Areas Commun. 40 (1) (2022) 308-322.

[10] G. Qiao, S. Leng, S. Maharjan, Y. Zhang, N. Ansari, Deep reinforcement learning for cooperative content caching in vehicular edge computing and networks, IEEE Internet Things J. 7 (1) (2020) 247-257.

[11] L. Cui, E. Ni, Y. Zhou, Z. Wang, L. Zhang, J. Liu, Y. Xu, Towards real-time video caching at edge servers: a cost-aware deep q-learning solution, IEEE Trans. Mul-timed. 25 (2023) 302-314.

[12] Y. Guo, F.R. Yu, J. An, K. Yang, C. Yu, V.C.M. Leung, Adaptive bitrate streaming in wireless networks with transcoding at network edge using deep reinforcement learning, IEEE Trans. Veh. Technol. 69 (4) (2020) 3879-3892.

[13] D. Wu, R. Bao, Z. Li, H. Wang, H. Zhang, R. Wang, Edge-cloud collaboration en-abled video service enhancement: a hybrid human-artificial intelligence scheme, IEEE Trans. Multimed. 23 (2021) 2208-2221.

[14] Z. Li, X. Gao, Q. Li, J. Guo, B. Yang, Edge caching enhancement for industrial In-ternet: a recommendation-aided approach, IEEE Internet Things J. 9 (18) (2022) 16941-16952.

[15] Y. Chiang, C.-H. Hsu, H.-Y. Wei, Collaborative social-aware and QoE-driven video caching and adaptation in edge network, IEEE Trans. Multimed. 23 (2021) 4311-4325.

[16] Y. Li, X. Wang, J. Wang, W. Wang, X. Luo, S. Xie, Cooperative multi-agent reinforce-ment learning with hierarchical relation graph under partial observability, in: Proc. IEEE 32nd Int. Conf. Tools Artif. Intell., 2020, pp. 1-8.

[17] P. Ammanabrolu, M. Riedl, Playing text-adventure games with graph-based deep reinforcement learning, Am. Chapter Assoc. Comput.in: NAACL HLT-Conf,in:N. Linguistics: Hum. Lang. Technol.-Proc. Conf., 2019, pp. 3557-3565.

[18] F. Zeng, K. Zhang, L. Wu, J. Wu, Efficient caching in vehicular edge computing based on edge-cloud collaboration, IEEE Trans. Veh. Technol. 72 (2) (2023) 2468-2481.

[19] Z. Ning, K. Zhang, X. Wang, L. Guo, X. Hu, J. Huang, B. Hu, R.Y.K. Kwok, Intelligent edge computing in Internet of vehicles: a joint computation offloading and caching solution, IEEE Trans. Intell. Transp. Syst. 22 (4) (2021) 2212-2225.

[20] X. Wang, C. Wang, X. Li, V.C.M. Leung, T. Taleb, Federated deep reinforcement learning for Internet of things with decentralized cooperative edge caching, IEEE Internet Things J. 7 (10) (2020) 9441-9455.

[21] J. Shi, L. Pu, J. Xu, Allies: tile-based joint transcoding, delivery and caching of 360° videos in edge cloud networks, in: Proc. IEEE 13th Int. Conf. Cloud Comput., 2020, pp. 337-344.

[22] A. Asheralieva, Optimal computational offloading and content caching in wireless heterogeneous mobile edge computing systems with Hopfield neural networks, IEEE Trans. Emerg. Top.comput. Intell. 5 (3) (2021) 407-425.

[23] Y. Qian, R. Wang, J. Wu, B. Tan, H. Ren, Reinforcement learning-based optimal computing and caching in mobile edge network, IEEE J. Sel. Areas Commun. 38 (10) (2020) 2343-2355.

[24] V. Kirilin, A. Sundarrajan, S. Gorinsky, R.K. Sitaraman, RL-cache: learning-based cache admission for content delivery, IEEE J. Sel. Areas Commun. 38 (10) (2020) 2372-2385.

[25] X. Zhao, S. Zhang, W. Dou, Multi-request scheduling and collaborative service pro-cessing for DASH-video optimization in cloud-edge network, in: Proc. IEEE 13th Int. Conf. Cloud Comput., 2020, pp. 582-589.

[26] P. Lin, Q. Song, A. Jamalipour, Multidimensional cooperative caching in CoMP-integrated ultra-dense cellular networks, IEEE Trans. Wirel.commun. 19 (3) (2020) 1977-1989.

[27] Y. He, N. Zhao, H. Yin, Integrated networking, caching, and computing for connected vehicles: a deep reinforcement learning approach, IEEE Trans. Veh. Technol. 67 (1) (2018) 44-55.

[28] F. Jiang, Y. Li, C. Sun, C. Wang, Dueling double deep q-network based computation offloading and resource allocation scheme for Internet of vehicles, in: 2023 IEEE Wireless Communications and Networking Conference (WCNC), 2023, pp. 1-6.

[29] Z. Gao, L. Yang, Y. Dai, Fast adaptive task offloading and resource allocation in large-scale MEC systems via multi-agent graph reinforcement learning, IEEE Internet Things J. 11 (1) (2023) 758-776.

[30] M. Schlichtkrull, T. Kipf, P. Bloem, R. van den Berg, I. Titov,M. Welling, Modeling relational data with graph convolutional networks, arXiv:1703.06103.

[31] T.X. Tran, D. Pompili, Adaptive bitrate video caching and processing in mobile-edge computing networks, IEEE Trans. Mob.comput. 18 (9) (2019) 1965-1978.