Learning-based cooperative content caching and sharing for multi-layer vehicular networks

Abstract

Caching and sharing the content files are critical and fundamental for various future vehicular applications. However, how to satisfy the content demands in a timely manner with limited storage is an open issue owing to the high mobility of vehicles and the unpredictable distribution of dynamic requests. To better serve the requests from the vehicles, a cache-enabled multi-layer architecture, consisting of a Micro Base Station (MBS) and several Small Base Stations (SBSs), is proposed in this paper. Considering that vehicles usually travel through the coverage of multiple SBSs in a short time period, the cooperative caching and sharing strategy is introduced, which can provide comprehensive and stable cache services to vehicles. In addition, since the content popularity profile is unknown, we model the content caching problems in a Multi-Armed Bandit (MAB) perspective to minimize the total delay while gradually estimating the popularity of content files. The reinforcement learning-based algorithms with a novel Q-value updating module are employed to update the caching files in different timescales for MBS and SBSs, respectively. Simulation results show the proposed algorithm outperforms benchmark algorithms with static or varying content popularity. In the high-speed environment, the cooperation between SBSs effectively improves the cache hit rate and further improves service performance.

Keywords

Cooperative content caching / MABReinforcement learning / Multi-layer vehicular networks / High-speed environment

CRediT authorship contribution statement

Jun Shi: Writing - original draft, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Yuanzhi Ni: Writing - review & editing, Supervision, Resources, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization. Lin Cai: Writing - review & editing, Supervision, Methodology. Zhuocheng Du: Writing - review & editing, Software, Data curation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the Natural Science Foundation of China (62002138) and the Open Research Project of the State Key Laboratory of Industrial Control Technology, Zhejiang University, China (ICT2022B26).

References

[1] G.S. Paschos, G. Iosifidis, M. Tao, D. Towsley, G. Caire, The role of caching in future communication systems and networks, IEEE J. Sel. Areas Commun. 36 (2018) 1111-1125.

[2] C. Fan, X. Zhou, T. Zhang, W. Yi, Y. Liu, Cache-enabled uav emergency communication networks: Performance analysis with stochastic geometry, IEEE Trans. Veh. Technol. (2023).

[3] X. Zhu, C. Jiang, L. Kuang, Z. Zhao, Cooperative multilayer edge caching in integrated satellite-terrestrial networks, IEEE Trans. Wireless Commun. 21 (2021) 2924-2937.

[4] A. Sadeghi, F. Sheikholeslami, G.B. Giannakis, Optimal and scalable caching for 5 g using reinforcement learning of space-time popularities, IEEE J. Sel. Top. Sign. Proces. 12 (2017) 180-190.

[5] A. Sadeghi, F. Sheikholeslami, A.G. Marques, G.B. Giannakis, Reinforcement learning for adaptive caching with dynamic storage pricing, IEEE J. Sel. Areas Commun. 37 (2019) 2267-2281.

[6] L. Lei, L. You, G. Dai, T.X. Vu, D. Yuan, S. Chatzinotas, A deep learn-ing approach for optimizing content delivering in cache-enabled hetnet, in: 2017 International Symposium on Wireless Communication Systems, ISWCS, IEEE, 2017, pp. 449-453.

[7] Y. He, Z. Zhang, F.R. Yu, N. Zhao, H. Yin, V.C. Leung, Y. Zhang, Deep-reinforcement-learning-based optimization for cache-enabled op-portunistic interference alignment wireless networks, IEEE Trans. Veh. Technol. 66 (2017) 10433-10445.

[8] X. Zhou, Z. Liu, M. Guo, J. Zhao, J. Wang, SACC: A size adaptive con-tent caching algorithm in fog/edge computing using deep reinforcement learning, IEEE Trans. Emerg. Top. Comput. 10 (2021) 1810-1820.

[9] C. Zhong, M.C. Gursoy, S. Velipasalar, A deep reinforcement learning-based framework for content caching, in: 2018 52nd Annual Conference on Information Sciences and Systems, CISS, IEEE, 2018, pp. 1-6.

[10] Y. He, N. Zhao, H. Yin, Integrated networking, caching, and computing for connected vehicles: A deep reinforcement learning approach, IEEE Trans. Veh. Technol. 67 (2017) 44-55.

[11] A. Sadeghi, G. Wang, G.B. Giannakis, Deep reinforcement learning for adaptive caching in hierarchical content delivery networks, IEEE Trans. Cogn. Commun. Netw. 5 (2019) 1024-1033.

[12] M. Dehghan, B. Jiang, A. Seetharam, T. He, T. Salonidis, J. Kurose, D. Towsley, R. Sitaraman, On the complexity of optimal request routing and content caching in heterogeneous cache networks, IEEE/ACM Trans. Netw. 25 (2016) 1635-1648.

[13] L. Tong, Y. Li, W. Gao, A hierarchical edge cloud architecture for mobile computing, in: IEEE INFOCOM 2016-the 35th Annual IEEE International Conference on Computer Communications, IEEE, 2016, pp. 1-9.

[14] S. Shukla, O. Bhardwaj, A.A. Abouzeid, T. Salonidis, T. He, Proac-tive retention-aware caching with multi-path routing for wireless edge networks, IEEE J. Sel. Areas Commun. 36 (2018) 1286-1299.

[15] W. Jiang, G. Feng, S. Qin, T.S.P. Yum, G. Cao, Multi-agent reinforcement learning for efficient content caching in mobile d2d networks, IEEE Trans. Wireless Commun. 18 (2019) 1610-1622.

[16] X. Xu, M. Tao, C. Shen, Collaborative multi-agent multi-armed bandit learning for small-cell caching, IEEE Trans. Wireless Commun. 19 (2020) 2570-2585.

[17] X. Xu, M. Tao, Decentralized multi-agent multi-armed bandit learning with calibration for multi-cell caching, IEEE Trans. Commun. 69 (2020) 2457-2472.

[18] X. Peng, J.-C. Shen, J. Zhang, K.B. Letaief, Backhaul-aware caching placement for wireless networks, in: 2015 IEEE Global Communications Conference, GLOBECOM, IEEE, 2015, pp. 1-6.

[19] K. Shanmugam, N. Golrezaei, A.G. Dimakis, A.F. Molisch, G. Caire, Fem-tocaching: Wireless content delivery through distributed caching helpers, IEEE Trans. Inform. Theory 59 (2013) 8402-8413.

[20] Y. Chen, M. Ding, J. Li, Z. Lin, G. Mao, L. Hanzo, Probabilistic small-cell caching: Performance analysis and optimization, IEEE Trans. Veh. Technol. 66 (2016) 4341-4354.

[21] A. Tang, S. Roy, X. Wang, Coded caching for wireless backhaul networks with unequal link rates, IEEE Trans. Commun. 66 (2017) 1-13.

[22] P. Blasco, D. Gündüz, Learning-based optimization of cache content in a small cell base station, in: 2014 IEEE International Conference on Communications, ICC, IEEE, 2014, pp. 1897-1903.

[23] J. Song, M. Sheng, T.Q. Quek, C. Xu, X. Wang, Learning-based content caching and sharing for wireless networks, IEEE Trans. Commun. 65 (2017) 4309-4324.

[24] W. Chen, Y. Wang, Y. Yuan, Combinatorial multi-armed bandit: General framework and applications,in:International Conference on Machine Learning, PMLR, 2013, pp. 151-159.

[25] S. Kapetanakis, D. Kudenko, M.J. Strens, Reinforcement learning approaches to coordination in cooperative multi-agent systems, in: Symposium on Adaptive Agents and Multi-Agent Systems, Springer, 2001, pp. 18-32.

[26] C. Claus, C. Boutilier, The dynamics of reinforcement learning in cooperative multiagent systems, in: AAAI/IAAI 1998, 1998, p. 2.

[27] L. Breslau, P. Cao, L. Fan, G. Phillips, S. Shenker, Web caching and zipf-like distributions: Evidence and implications, in: IEEE INFOCOM’99, vol. 1, IEEE, 1999, pp. 126-134.

[28] A. Korbut, I.K. Sigal, Exact and greedy solutions of the knapsack problem: The ratio of values of objective functions, J. Comput. Syst. Sci. Int. 49 (2010) 757-764.

[29] S. Müller, O. Atan, M. van der Schaar, A. Klein, Context-aware proactive content caching with service differentiation in wireless networks, IEEE Trans. Wireless Commun. 16 (2016) 1024-1036.

[30] G. Zhang, B. Tang, X. Wang, Y. Wu, An optimal cache placement strategy based on content popularity in content centric network, J. Inf. Comput. Sci. 11 (2014) 2759-2769.

[31] M. Bilal, S.-G. Kang, Time aware least recent used (TLRU) cache man-agement policy in ICN, in: 16th International Conference on Advanced Communication Technology, IEEE, 2014, pp. 528-532.