Energy-saving control strategy for ultra-dense network base stations based on multi-agent reinforcement learning✩

Yan Zhen; Litianyi Tao; Dapeng Wu; Tong Tang; Ruyan Wang

doi:10.1016/j.dcan.2024.10.015

›› 2025, Vol. 11 ›› Issue (4) :1007 -1017. DOI: 10.1016/j.dcan.2024.10.015

Research article

research-article

Energy-saving control strategy for ultra-dense network base stations based on multi-agent reinforcement learning^✩

Yan Zhen ^a^,^b^,^c^,^d^,^*
, Litianyi Tao ^a^,^c^,^d
, Dapeng Wu ^a^,^c^,^d
, Tong Tang ^a^,^c^,^d
, Ruyan Wang ^a^,^c^,^d

Author information +

History +

PDF

Abstract

Aiming at the problem of mobile data traﬃc surge in 5G networks, this paper proposes an eﬀective solution combining massive multiple-input multiple-output techniques with Ultra-Dense Network (UDN) and focuses on solving the resulting challenge of increased energy consumption. A base station control algorithm based on Multi-Agent Proximity Policy Optimization (MAPPO) is designed. In the constructed 5G UDN model, each base station is considered as an agent, and the MAPPO algorithm enables inter-base station collaboration and interference management to optimize the network performance. To reduce the extra power consumption due to frequent sleep mode switching of base stations, a sleep mode switching decision algorithm is proposed. The algorithm reduces unnecessary power consumption by evaluating the network state similarity and intelligently adjusting the agent’s action strategy. Simulation results show that the proposed algorithm reduces the power consumption by 24.61% compared to the no-sleep strategy and further reduces the power consumption by 5.36% compared to the traditional MAPPO algorithm under the premise of guaranteeing the quality of service of users.

Keywords

Ultra dense networks / Base station sleep / Multiple input multiple output / Reinforcement learning

Cite this article

Download citation ▾

Yan Zhen, Litianyi Tao, Dapeng Wu, Tong Tang, Ruyan Wang. Energy-saving control strategy for ultra-dense network base stations based on multi-agent reinforcement learning^✩. , 2025, 11(4): 1007-1017 DOI:10.1016/j.dcan.2024.10.015

登录浏览全文

4963

注册一个新账户忘记密码

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	X. Qiu, C.-Y. Chiang, P. Lin, S.-R. Yang, C.-W. Huang, Reinforcement learning-based network management based on son for the 5G mobile network, in: 2023 Inter-national Wireless Communications and Mobile Computing (IWCMC), IEEE, 2023, pp. 1503-1508.

[2]	F. Han, S. Zhao, L. Zhang, J. Wu, Survey of strategies for switching oﬀ base stations in heterogeneous networks for greener 5G systems, IEEE Access 4 (2016) 4959-4973.

[3]	T. Sizer, D. Samardzija, H. Viswanathan, S.T. Le, S. Bidkar, P. Dom, E. Harstead, T. Pfeiﬀer, Integrated solutions for deployment of 6G mobile networks, J. Lightwave Technol. 40 (2) (2021) 346-357.

[4]	J. Wu, Green wireless communications: from concept to reality [industry perspec-tives], IEEE Wirel. Commun. 19 (4) (2012) 4-5.

[5]	A. Jee, S. Prakriya, Performance of energy and spectrally eﬃcient af relay-aided incremental cdrt noma-based iot network with imperfect sic for smart cities, IEEE Int. Things J. 10 (21) (2022) 18766-18781.

[6]	A. Jee, K. Agrawal, S. Prakriya, A coordinated direct af/df relay-aided noma frame-work for low outage, IEEE Trans. Commun. 70 (3) (2021) 1559-1579.

[7]	A. Jee, S. Prakriya, Performance of statistical power control with simplified user selection and mode switching in hybrid noma/oma networks, IEEE Trans. Green Commun. Netw. (2024) 1-11.

[8]	S. Buzzi, I. Chih-Lin, T.E. Klein, H.V. Poor, C. Yang, A. Zappone, A survey of energy-eﬃcient techniques for 5G networks and challenges ahead, IEEE J. Sel. Areas Com-mun. 34 (4) (2016) 697-709.

[9]	C. Liu, B. Natarajan, H. Xia, Small cell base station sleep strategies for energy eﬃ-ciency, IEEE Trans. Veh. Technol. 65 (3) (2015) 1652-1661.

[10]	J. Wu, S. Guo, H. Huang, W. Liu, Y. Xiang, Information and communications tech-nologies for sustainable development goals: state-of-the-art, needs and perspectives, IEEE Commun. Surv. Tutor. 20 (3) (2018) 2389-2406.

[11]	Z. Wang, T. Lv, W. Li, Energy eﬃciency maximization in massive mimo-noma net-works with non-linear energy harvesting, in: 2021 IEEE Wireless Communications and Networking Conference (WCNC), IEEE, 2021, pp. 1-6.

[12]	S. Marwaha, E.A. Jorswieck, D. López-Pérez, X. Geng, H. Bao, Spatial and spectral resource allocation for energy-eﬃcient massive mimo 5G networks, in: ICC 2022-IEEE International Conference on Communications, IEEE, 2022, pp. 135-140.

[13]

I.L.S.committee, et al., IEEE standard for local and metropolitan area networks part 16: air interface for fixed and mobile broadband wireless access systems amend-ment 2: Physical and medium access control layers for combined fixed and mobile operation in licensed bands and corrigendum 1, IEEE Std 802.16 e-2005.

[14]	M. Ozturk, A.I. Abubakar, J.P.B. Nadas, R.N.B. Rais, S. Hussain, M.A. Imran, Energy optimization in ultra-dense radio access networks via traﬃc-aware cell switching, IEEE Trans. Green Commun. Netw. 5 (2) (2021) 832-845.

[15]	A. El Amine, P. Dini, L. Nuaymi, Reinforcement learning for delay-constrained energy-aware small cells with multi-sleeping control, in: 2020 IEEE International Conference on Communications Workshops (ICC Workshops), IEEE, 2020, pp. 1-6.

[16]	S. Malta, P. Pinto, M. Fernández-Veiga, Using reinforcement learning to reduce en-ergy consumption of ultra-dense networks with 5G use cases requirements, IEEE Access 11 (2023) 5417-5428.

[17]	J. Miao, X. Gu, B. Leng, Energy eﬃciency analysis for future network with inter-base station cooperation based sleep strategy, in: 2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications, IEEE, 2020, pp. 1-6.

[18]	L. Li, W. Meng, Collaborative base station sleeping solution design in heterogeneous cellular network, in: 2022 27th Asia Pacific Conference on Communications (APCC), IEEE, 2022, pp. 231-235.

[19]	K.-C. Chang, K.-C. Chu, H.-C. Wang, Y.-C. Lin, J.-S. Pan, Energy saving technology of 5G base station based on Internet of Things collaborative control, IEEE Access 8 (2020) 32935-32946.

[20]	G. Chopra, An eﬃcient base station sleeping configuration for ultra-dense networks, in: 2023 International Conference on Emerging Smart Computing and Informatics (ESCI), IEEE, 2023, pp. 1-5.

[21]	D. Sudhakar, D. Sunehra, A traﬃc-aware and power-saving base station switch oﬀ method for cellular networks, in: 2022 7th International Conference on Communi-cation and Electronics Systems (ICCES), IEEE, 2022, pp. 12-15.

[22]	J.-H. Noh, B. Lee, S.-J. Oh, User-number threshold-based base station on/oﬀ con-trol for maximizing coverage probability, IEEE Trans. Veh. Technol. 71 (3) (2022) 3214-3228.

[23]	X. Tan, K. Xiong, B. Gao, P. Fan, K.B. Letaief, Energy-eﬃcient base station switching-oﬀ with guaranteed cooperative profit gain of mobile network operators, IEEE Trans. Green Commun. Netw. 7 (3) (2023) 1250-1266.

[24]	Z. Shen, G. Zhang, Two-timescale mobile user association and hybrid generator on/oﬀ control for green cellular networks with energy storage, IEEE Trans. Veh. Technol. 71 (10) (2022) 11047-11059.

[25]	W. Lee, H. Lee, H.-H. Choi, Deep learning-based network-wide energy eﬃciency optimization in ultra-dense small cell networks, IEEE Trans. Veh. Technol. 72 (6) (2023) 8244-8249.

[26]	Z. Guo, Z. Chen, P. Liu, J. Luo, X. Yang, X. Sun, Multi-agent reinforcement learning-based distributed channel access for next generation wireless networks, IEEE J. Sel. Areas Commun. 40 (5) (2022) 1587-1599.

[27]	E.E. Tsiropoulou, P. Vamvakas, S. Papavassiliou, Joint utility-based uplink power and rate allocation in wireless networks: a non-cooperative game theoretic frame-work, Phys.commun. 9 (2013) 299-307.

[28]	B. Debaillie, C. Desset, F. Louagie, A flexible and future-proof power model for cellu-lar base stations, in: 2015 IEEE 81st Vehicular Technology Conference, (VTC Spring), IEEE, 2015, pp. 1-7.

[29]	D. Kakadia, J. Yang, A. Gilgur, D. Kakadia, J. Yang, A. Gilgur, Evolved universal ter-restrial radio access network (eutran), in: Network Performance and Fault Analytics for LTE Wireless Service Providers, 2017, pp. 61-81.

[30]	D. López-Pérez, A. De Domenico, N. Piovesan, G. Xinli, H. Bao, S. Qitao, M. Debbah, A survey on 5G radio access network energy eﬃciency: massive mimo, lean car-rier design, sleep modes, and machine learning, IEEE Commun. Surv. Tutor. 24 (1) (2022) 653-697.

[31]	W. Li, M. Latva-aho, An eﬃcient channel block diagonalization method for general-ized zero forcing assisted mimo broadcasting systems, IEEE Trans. Wirel.commun. 10 (3) (2010) 739-744.

[32]	M.A. Hossain, C. Cavdar, E. Bjornson, R. Jantti, Energy-eﬃcient load-adaptive mas-sive mimo, in: 2015 IEEE Globecom Workshops (GC Wkshps), IEEE, 2015, pp. 1-6.

[33]	Q. Cui, Y. Zhang, W. Ni, M. Valkama, R. Jäntti, Energy eﬃciency maximization of full-duplex two-way relay with non-ideal power amplifiers and non-negligible circuit power, IEEE Trans. Wirel.commun. 16 (9) (2017) 6264-6278.

[34]	M.M.A. Hossain, K. Koufos, R. Jantti, Minimum-energy power and rate control for fair scheduling in the cellular downlink under flow level delay constraint, IEEE Trans. Wirel.commun. 12 (7) (2013) 3253-3263.

[35]	S. Tombaz, P. Frenger, M. Olsson, A. Nilsson, Energy performance of 5G-NX radio access at country level, in: 2016 IEEE 12th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), IEEE, 2016, pp. 1-6.

[36]	M. Haklidir, H. Temeltaş, Guided soft actor critic: a guided deep reinforcement learning approach for partially observable Markov decision processes, IEEE Access 9 (2021) 159672-159683.

[37]	H. Kang, X. Chang, J. Mišić, V.B. Mišić, J. Fan, Y. Liu, Cooperative UAV resource allocation and task oﬄoading in hierarchical aerial computing systems: a mappo based approach, IEEE Int. Things J. 10 (12) (2023) 10497-10509.

[38]	D. Guo, L. Tang, X. Zhang, Y.-C. Liang, Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning, IEEE Trans. Veh. Technol. 69 (11) (2020) 13124-13138.

[39]	J. Wu, Y. Zhang, M. Zukerman, E.K.-N. Yung, Energy-eﬃcient base-stations sleep-mode techniques in green cellular networks: a survey, IEEE Commun. Surv. Tutor. 17 (2) (2015) 803-826.

[40]	X. Lin, Y. Tang, B. Zhou, Improved model predictive control path tracking strategy based an online updating algorithm with cosine similarity and a horizon factor, IEEE Trans. Intell. Transp. Syst. 23 (8) (2021) 12429-12438.

[41]	H. Liu, Y. Li, S. Wang, Request scheduling combined with load balancing in mobile-edge computing, IEEE Int. Things J. 9 (21) (2022) 20841-20852.

[42]	D. Kato, Y. Muroki, N. Nonaka, K. Higuchi,Activation control of base stations based on multi-agent dqn for heterogeneous networks, in: 2022 IEEE 96th Vehicular Tech-nology Conference (VTC2022-Fall), IEEE, 2022, pp. 1-5.

[43]	A.A. Razzac, T. Chahed, Z. Shamseddine, W. Zahwa,Advanced sleep modes in 5G multiple base stations using non-cooperative multi-agent reinforcement learning, in: GLOBECOM 2023-2023 IEEE Global Communications Conference, IEEE, 2023, pp. 7025-7030.