Reinforcement learning based edge computing in B5G

Abstract

The development of communication technology will promote the application of Internet of Things, and Beyond 5G will become a new technology promoter. At the same time, Beyond 5G will become one of the important supports for the development of edge computing technology. This paper proposes a communication task allocation algorithm based on deep reinforcement learning for vehicle-to-pedestrian communication scenarios in edge computing. Through trial and error learning of agent, the optimal spectrum and power can be determined for transmission without global information, so as to balance the communication between vehicle-to-pedestrian and vehicle-to-infrastructure. The results show that the agent can effectively improve vehicle-to-infrastructure communication rate as well as meeting the delay constraints on the vehicle-to-pedestrian link.

Keywords

Reinforcement learning / Edge computing / Beyond 5G / Vehicle-to-pedestrian

References

[1] Z. Lv, A.K. Singh, J. Li, Deep learning for security problems in 5g heterogeneous networks, IEEE Network 35 (2) (2021) 67-73.

[2] Q. Qi, X. Chen, C. Zhong, Z. Zhang, Integrated sensing, computation and communication in b5g cellular internet of things, IEEE Trans. Wireless Commun. 20 (1) (2021) 332-344.

[3] M. Chen, X. Wei, J. Chen, L. Wang, L. Zhou, Integration and provision for city public service in smart city cloud union: architecture and analysis, IEEE Wireless Commun. (99) (2020) 1-7.

[4] D. Jiang, Z. Wang, W. Wang, Z. Lv, K.-K.R. Choo, Ai-assisted energy-efficient and intelligent routing for reconfigurable wireless networks, IEEE Trans. Netw. Sci. Eng. 9 (1) (2022) 78-88.

[5] M. Chen, L. Wang, J. Chen, X. Wei, L. Lei, A computing and content delivery network in the smart city: scenario, framework, and analysis, IEEE Network 33 (2)(2019) 89-95.

[6] H. Tabassum, M. Salehi, E. Hossain,Mobility-aware Analysis of 5g and B5g Cellular Networks: A Tutorial, 2018 arXiv preprint arXiv: 1805.02719.

[7] J. Yang, J. Wen, Y. Wang, B. Jiang, H. Wang, H. Song, Fog-based marine environmental information monitoring toward ocean of things, IEEE Internet Things J. 7 (5) (2020) 4238-4247.

[8] Y. Li, S. Wang, An energy-aware edge server placement algorithm in mobile edge computing, in: 2018 IEEE International Conference on Edge Computing (EDGE), IEEE, 2018, pp. 66-73.

[9] G. Gao, Y. Wen, Video transcoding for adaptive bitrate streaming over edge-cloud continuum, Digit. Commun. Networks 7 (4) (2021) 598-604.

[10] M. Dibaei, X. Zheng, K. Jiang, R. Abbas, S. Yu, Attacks and defences on intelligent connected vehicles: a survey, Digit. Commun. Networks 6 (4) (2020) 399-421.

[11] D. Jiang, P. Zhang, Z. Lv, H. Song, Energy-efficient multi-constraint routing algorithm with load balancing for smart city applications, IEEE Internet Things J. 3 (6) (2016) 1437-1447.

[12] L. Melki, S. Najeh, H. Besbes, Radio resource management scheme and outage analysis for network-assisted multi-hop d2d communications, Digit. Commun. Networks 2 (4) (2016) 225-232.

[13] K.U. A, C.D. A, A cognitive v2v communication system model using active user cooperation in 3d-gbsm channel, in: 2020 4th International Conference on Computer, Communication and Signal Processing (ICCCSP), IEEE, 2020, pp. 1-4.

[14] J. Yang, J. Wen, B. Jiang, H. Wang, Blockchain-based sharing and tamper-proof framework of big data networking, IEEE Network 34 (4) (2020) 62-67.

[15] Y. Li, X. Chao, Semi-supervised few-shot learning approach for plant diseases recognition, Plant Methods 17 (2021), 68.

[16] Y. Li, J. Yang, Meta-learning baselines and database for few-shot classification in agriculture, Comput. Electron. Agric. 182 (5) (2021), 106055.

[17] J. Yang, Y. Zhao, J. Liu, B. Jiang, X. Gao, No reference quality assessment for screen content images using stacked autoencoders in pictorial and textual regions, IEEE Trans. Cybern. (99) (2020) 1-13.

[18] W. Chen, X. Qiu, T. Cai, H.-N. Dai, Z. Zheng, Y. Zhang, Deep reinforcement learning for internet of things: a comprehensive survey, IEEE Commun. Surv. Tutor. 23 (3)(2021) 1659-1692.

[19] H. Fei, Y. Zhang, Y. Ren, D. Ji, Optimizing attention for sequence modeling via reinforcement learning, IEEE Transact. Neural Networks Learn. Syst. (2021) 1-10.

[20] K. Sim, J. Yang, W. Lu, X. Gao, Mad-dls: Mean and deviation of deep and local similarity for image quality assessment, IEEE Trans. Multimed. 23 (2020) 4037-4048.

[21] S. Gangapurwala, A. Mitchell, I. Havoutis, Guided constrained policy optimization for dynamic quadrupedal robot locomotion, IEEE Robot. Autom. Lett. 5 (2) (2020) 3642-3649.

[22] N. Brown, T. Sandholm, Safe and nested subgame solving for imperfect-information games, in: Proceedings of 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA (2017) 689-699.

[23] T. Teichmann, M.J. Gonzalez Torres, K. Makarevich, S. Polter, P. Lachmann, E.R. van der Graaf, M.J. van Goethem, A. Jahn, J. Henniger, K. Zuber, T. Kormoll, Combined osl-rl measurements for dosimetry in mixed let proton fields, in: 2019 IEEE Nuclear Science Symposium and Medical Imaging Conference, NSS/MIC, 2019, pp. 1-3.

[24] J. Yang, J. Zhang, H. Wang, Urban traffic control in software defined internet of things via a multi-agent deep reinforcement learning approach, IEEE Trans. Intell. Transport. Syst. 22 (6) (2020) 1-13.

[25] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al., Qt-opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation, 2018 arXiv preprint arXiv: 1806.10293.

[26] J. Jin, C. Song, H. Li, K. Gai, J. Wang, W. Zhang, Real-time bidding with multi-agent reinforcement learning in display advertising, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, ACM, 2018, pp. 2193-2201.

[27] C. Yu, J. Liu, S. Nemati, Reinforcement Learning in Healthcare: A Survey, 2019 arXiv preprint arXiv: 1908.08796.

[28] X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y.-F. Wang, W.Y. Wang, L. Zhang, Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2019, pp. 6629-6638.

[29] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller, Playing atari with deep reinforcement learning, Comput. Sci. 21 (2013) 351-362.

[30] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, D. Hassabis, Mastering the game of go without human knowledge, Nature 550 (7676) (2017) 354-359.

[31] O. Vinyals, I. Babuschkin, W.M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al., Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature 575 (7782) (2019) 350-354.

[32] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al., Dota 2 with Large Scale Deep Reinforcement Learning, 2019 arXiv preprint arXiv: 1912.06680.

[33] B. Wu,Hierarchical macro strategy model for moba game ai, in:Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 1206-1213.

[34] Y. Li, J. Yang, Few-shot cotton pest recognition and terminal realization, Comput. Electron. Agric. 169 (2020), 105240.

[35] Z. Liu, X. Yin, Y. Hu, Cpss lr-ddos detection and defense in edge computing utilizing dcnn q-learning, IEEE Access 8 (2020) 42120-42130.

[36] Y. Chen, S. Deng, H. Zhao, Q. He, Y. Li, H. Gao, Data-intensive application deployment at edge: a deep reinforcement learning approach, in: 2019 IEEE International Conference on Web Services (ICWS), IEEE, 2019, pp. 355-359.

[37] Z. Ning, P. Dong, X. Wang, J.J. Rodrigues, F. Xia, Deep reinforcement learning for vehicular edge computing: an intelligent offloading system, ACM Trans. Intell. Syst. Technol. (TIST) 10 (6) (2019) 1-24.

[38] Q. Jin, S. Ge, J. Zeng, X. Zhou, T. Qiu, Scarl: service function chain allocation based on reinforcement learning in mobile edge computing, in: 2019 Seventh International Conference on Advanced Cloud and Big Data (CBD), IEEE, 2019, pp. 327-332.

[39] S. Xia, Z. Yao, Y. Li, S. Mao, Online distributed offloading and computing resource management with energy harvesting for heterogeneous mec-enabled iot, IEEE Trans. Wireless Commun. 20 (10) (2021) 6743-6757.

[40] Y. Li, H. Ma, L. Wang, S. Mao, G. Wang, Optimized content caching and user association for edge computing in densely deployed heterogeneous networks, IEEE Transactions on Mobile Computing 21 (6) (2022) 1233-1536.

[41] J. Yang, C. Wang, B. Jiang, H. Song, Q. Meng, Visual perception enabled industry intelligence: state of the art, challenges and prospects, IEEE Trans. Ind. Inf. 17 (3)(2020) 2204-2219.

[42] Y. Li, J. Nie, X. Chao, Do we really need deep cnn for plant diseases identification? Comput. Electron. Agric. 178 (3) (2020), 105803.