Container cluster placement in edge computing based on reinforcement learning incorporating graph convolutional networks scheme

Abstract

Container-based virtualization technology has been more widely used in edge computing environments recently due to its advantages of lighter resource occupation, faster startup capability, and better resource utilization efficiency. To meet the diverse needs of tasks, it usually needs to instantiate multiple network functions in the form of containers interconnect various generated containers to build a Container Cluster (CC). Then CCs will be deployed on edge service nodes with relatively limited resources. However, the increasingly complex and time-varying nature of tasks brings great challenges to optimal placement of CC. This paper regards the charges for various resources occupied by providing services as revenue, the service efficiency and energy consumption as cost, thus formulates a Mixed Integer Programming (MIP) model to describe the optimal placement of CC on edge service nodes. Furthermore, an Actor-Critic based Deep Reinforcement Learning (DRL) incorporating Graph Convolutional Networks (GCN) framework named as RL-GCN is proposed to solve the optimization problem. The framework obtains an optimal placement strategy through self-learning according to the requirements and objectives of the placement of CC. Particularly, through the introduction of GCN, the features of the association relationship between multiple containers in CCs can be effectively extracted to improve the quality of placement. The experiment results show that under different scales of service nodes and task requests, the proposed method can obtain the improved system performance in terms of placement error ratio, time efficiency of solution output and cumulative system revenue compared with other representative baseline methods.

Keywords

Edge computing / Network virtualization / Container cluster / Deep reinforcement learning / Graph convolutional network

Declaration of competing interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled.

References

[1] P. Mach, Z. Becvar, Mobile edge computing: a survey on architecture and computation offloading, IEEE Commun. Surv. Tutorials 19 (3) (2017) 1628-1656.

[2] L. Zhang, B. Cao, Y. Li, M. Peng, G. Feng, A multi-stage stochastic programmingbased offloading policy for fog enabled iot-ehealth, IEEE J. Sel. Area. Commun 39 (2) (2020) 411-425.

[3] B. Cao, L. Zhang, Y. Li, D. Feng, W. Cao, Intelligent offloading in multi-access edge computing: a state-of-the-art review and framework, IEEE Commun. Mag. 57 (3) (2019) 56-62.

[4] M. Huang, W. Liang, X. Shen, Y. Ma, H. Kan, Reliability-aware virtualized network function services provisioning in mobile edge computing, IEEE Trans. Mobile Comput. 19 (11) (2020) 2699-2713.

[5] M. Li, Q. Zhang, F. Liu, Finedge: a dynamic cost-efficient edge resource management platform for nfv network,in:2020 IEEE/ACM 28th International Symposium on Quality of Service, IWQoS), 2020, pp. 1-10.

[6] Y. Li, S. Xia, M. Zheng, B. Cao, Q. Liu, Lyapunov optimization-based trade-off policy for mobile cloud offloading in heterogeneous wireless networks, IEEE. Trans. Cloud Comput. 10 (1) (2022) 491-505.

[7] Z. Tao, Q. Xia, Z. Hao, C. Li, L. Ma, S. Yi, Q. Li, A survey of virtual machine management in edge computing, Proc. IEEE 107 (8) (2019) 1482-1499.

[8] R. Xie, Q. Tang, S. Qiao, H. Zhu, F.R. Yu, T. Huang, When serverless computing meets edge computing: architecture, challenges, and open issues, IEEE Wireless Commun. 28 (5) (2021) 126-133.

[9] B. Sonkoly, J. Czentye, M. Szalay, B. N_emeth, L. Toka, Survey on placement methods in the edge and beyond, IEEE Commun. Surv. Tutorials 23 (4) (2021) 2590-2629.

[10] Optimized container scheduling for data-intensive serverless edge computing, Future Generat. Comput. Syst. 114 (2021) 259-271.

[11] Y. Wang, C.-K. Huang, S.-H. Shen, G.-M. Chiu, Adaptive placement and routing for service function chains with service deadlines, IEEE.Trans.Netw.Serv. Manag. 18 (3) (2021) 3021-3036.

[12] D. Guo, B. Ren, G. Tang, L. Luo, T. Chen, X. Fu, Optimal embedding of aggregated service function tree, IEEE Trans. Parallel Distr. Syst. 33 (10) (2022) 2584-2596.

[13] T.N. Kipf, M. Welling, Semi-supervised Classification with Graph Convolutional Networks, 2016 arXiv preprint arXiv:1609.02907.

[14] A. Varasteh, B. Madiwalar, A. Van Bemten, W. Kellerer, C. Mas-Machuca, Holu: power-aware and delay-constrained vnf placement and chaining, IEEE.Trans.Netw.Serv. Manag. 18 (2) (2021) 1524-1539.

[15] G.P. Sharma, W. Tavernier, D. Colle, M. Pickavet, Vnf-aapc: accelerator-aware vnf placement and chaining, Comput. Network. 177 (2020) 107329.

[16] C. Assi, S. Ayoubi, N. El Khoury, L. Qu, Energy-aware mapping and scheduling of network flows with deadlines on vnfs, IEEE Trans. Green.Commun. Netw. 3 (1) (2019) 192-204.

[17] Y. Mao, X. Shang, Y. Yang, Near-optimal resource allocation and virtual network function placement at network edges, in: 2021 IEEE 27th International Conference on Parallel and Distributed Systems, ICPADS), 2021, pp. 18-25.

[18] B. Mutichiro, Y. Kim, User preference-based qos-aware service function placement in iot-edge cloud, Int. J. Distributed Sens. Netw. 17 (5) (2021) 15501477211019912.

[19] Y. Li, F. Zheng, M. Chen, D. Jin, A unified control and optimization framework for dynamical service chaining in software-defined nfv system, IEEE Wireless Commun. 22 (6) (2015) 15-23.

[20] M. Karimzadeh-Farshbafan, V. Shah-Mansouri, D. Niyato, A dynamic reliabilityaware service placement for network function virtualization (nfv), IEEE J. Sel. Area. Commun. 38 (2) (2020) 318-333.

[21] H. Hawilo, M. Jammal, A. Shami, Network function virtualization-aware orchestrator for service function chaining placement in the cloud, IEEE J. Sel. Area. Commun. 37 (3) (2019) 643-655.

[22] S.K. Kasi, M.K. Kasi, K. Ali, M. Raza, H. Afzal, A. Lasebae, B. Naeem, S.u. Islam, J.J.P.C. Rodrigues, Heuristic edge server placement in industrial internet of things and cellular networks, IEEE Internet Things J. 8 (13) (2021) 10308-10317.

[23] P. Zhou, G. Wu, B. Alzahrani, A. Barnawi, A. Alhindi, M. Chen, Reinforcement learning for task placement in collaborative cloud-edge computing, in: 2021 IEEE Global Communications Conference (GLOBECOM), IEEE, 2021, pp. 1-6.

[24] Y. Mu, L. Wang, J. Zhao,Energy-efficient and interference-aware vnf placement with deep reinforcement learning, in:2021 IFIP Networking Conference (IFIP Networking), 2021, pp. 1-9.

[25] R. Solozabal, J. Ceberio, A. Sanchoyerto, L. Zabala, B. Blanco, F. Liberal, Virtual network function placement optimization with deep reinforcement learning, IEEE J. Sel. Area. Commun. 38 (2) (2019) 292-303.

[26] T. Wang, Q. Fan, X. Li, X. Zhang, Q. Xiong, S. Fu, M. Gao,Drl-sfcp: adaptive service function chains placement with deep reinforcement learning, in: ICC 2021 - IEEE International Conference on Communications, 2021, pp. 1-6.

[27] Z. Zhang, H. Liu, M. Zhou, J. Wang, Solving dynamic traveling salesman problems with deep reinforcement learning, IEEE Transact. Neural Networks Learn. Syst. (2021) 1-14.

[28] T. Liu, S. Ni, X. Li, Y. Zhu, L. Kong, Y. Yang, Deep reinforcement learning based approach for online service placement and computation resource allocation in edge computing, IEEE Trans. Mobile Comput. 22 (7) (2023) 3870-3881.

[29] L. Gu, D. Zeng, J. Hu, B. Li, H. Jin,Layer aware microservice placement and request scheduling at the edge, in: IEEE INFOCOM 2021 - IEEE Conference on Computer Communications, 2021, pp. 1-9.

[30] Z. Huang, N. Samaan, A. Karmouch,A novel resource reliability-aware infrastructure manager for containerized network functions, in: ICC 2021 - IEEE International Conference on Communications, 2021, pp. 1-6.

[31] R.F. Ustok, U. Acar, S. Keskin, D. Breitgand, A. Weit, P. Drakoulis, A. Doumanoglou, N. Zioulis, D. Zarpalas, P. Daras, F. Iadanza, F. Moscatelli, G. Bernini, Service development kit for media-type virtualized network services in 5g networks, IEEE Commun. Mag. 58 (7) (2020) 51-57.

[32] A. Tootoonchian, A. Panda, C. Lan, M. Walls, K. Argyraki, S. Ratnasamy, S. Shenker, {ResQ}: enabling {SLOs} in network function virtualization,in:15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), 2018, pp. 283-297.

[33] H. Yu, Z. Zheng, J. Shen, C. Miao, C. Sun, H. Hu, J. Bi, J. Wu, J. Wang, Octans: optimal placement of service function chains in many-core systems, IEEE Trans. Parallel Distr. Syst. 32 (9) (2021) 2202-2215.

[34] T.H. Cormen, C.E. Leiserson, R.L. Rivest, C. Stein, Introduction to Algorithms, MIT press, 2022.

[35] T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous Control with Deep Reinforcement Learning, 2015 arXiv preprint arXiv: 1509.02971.

[36] I. Sutskever, O. Vinyals, Q.V. Le, Sequence to sequence learning with neural networks, Proceedings of the 28th International Conference on Neural Information Porcessing Systems, (2014) 3104-3112.

[37] T.N. Kipf, M. Welling, Variational Graph Auto-Encoders, 2016 arXiv preprint arXiv: 1611.07308.

[38] W. Jin, K. Yang, R. Barzilay, T. Jaakkola, Learning Multimodal Graph-To-Graph Translation for Molecular Optimization, 2018 arXiv preprint arXiv: 1812.01070.

[39] K. Hu, J. Wu, Y. Li, M. Lu, L. Weng, M. Xia, Fedgcn: federated learning-based graph convolutional networks for non-euclidean spatial data, Mathematics and computer Science 10 (6) (2022) 1000.

[40] T. Tanaka, H. Sandberg, M. Skoglund, Transfer-entropy-regularized markov decision processes, IEEE Trans. Automat. Control 67 (4) (2021) 1944-1951.

[41] TensorFlow: large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/, 2022. (Accessed 7 July 2022).