A novel handover scheme for millimeter wave network: An approach of integrating reinforcement learning and optimization

Abstract

The millimeter-Wave (mmWave) communication with the advantages of abundant bandwidth and immunity to interference has been deemed a promising technology to greatly improve network capacity. However, due to such characteristics of mmWave, as short transmission distance, high sensitivity to the blockage, and large propagation path loss, handover issues (including trigger condition and target beam selection) become much complicated. In this paper, we design a novel handover scheme to optimize the overall system throughput as well as the total system delay while guaranteeing the Quality of Service (QoS) of each User Equipment (UE). Specifically, the proposed handover scheme called O-MAPPO integrates the Reinforcement Learning (RL) algorithm and optimization theory. The RL algorithm known as Multi-Agent Proximal Policy Optimization (MAPPO) plays a role in determining handover trigger conditions. Further, we propose an optimization problem in conjunction with MAPPO to select the target base station. The aim is to evaluate and optimize the system performance of total throughput and delay while guaranteeing the QoS of each UE after the handover decision is made. The numerical results show the overall system throughput and delay with our method are slightly worse than that with the exhaustive search method but much better than that using another typical RL algorithm Deep Deterministic Policy Gradient (DDPG).

Keywords

Reinforcement learning / Optimization theory / MmWave / Cellular network / Handover

References

[1] W. Roh, J.-Y. Seol, J. Park, B. Lee, J. Lee, Y. Kim, J. Cho, K. Cheun, F. Aryanfar, Millimeter-wave beamforming as an enabling technology for 5g cellular commu-nications: theoretical feasibility and prototype results, IEEE Commun. Mag. 52 (2)(2014) 106-113.

[2] B.D. Van Veen, K.M. Buckley, Beamforming: a versatile approach to spatial filtering, IEEE ASSP Mag. 5(2) (1988) 4-24.

[3] T. Bai, R.W. Heath, Coverage and rate analysis for millimeter-wave cellular net-works, IEEE Trans. Wirel. Commun. 14 (2) (2014) 1100-1114.

[4] Q. Xue, Y. Sun, J. Wang, G. Feng, L. Yan, S. Ma, User-centric association in ultra-dense mmwave networks via deep reinforcement learning, IEEE Commun. Lett. 25 (11) (2021) 3594-3598.

[5] M. Tayyab, X. Gelabert, R. Jäntti, A survey on handover management: from lte to nr, IEEE Access 7 (2019) 118907-118930.

[6] Y. Sun, W. Jiang, G. Feng, P.V. Klaine, L. Zhang, M.A. Imran, Y.-C. Liang, Effi-cient handover mechanism for radio access network slicing by exploiting distributed learning, IEEE Trans. Netw. Serv. Manag. 17 (4) (2020) 2620-2633.

[7] D. Guo, L. Tang, X. Zhang, Y.-C. Liang, Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning, IEEE Trans. Veh. Technol. 69 (11) (2020) 13124-13138.

[8] Y. Sun, G. Feng, S. Qin, Y.-C. Liang, T.-S.P. Yum, The smart handoff policy for millimeter wave heterogeneous cellular networks, IEEE Trans. Mob. Comput. 17 (6)(2017) 1456-1468.

[9] B. Van Quang, R.V. Prasad, I. Niemegeers, A survey on handoffs—lessons for 60 ghz based wireless systems, IEEE Commun. Surv. Tutor. 14 (1) (2010) 64-86.

[10] A. Talukdar, M. Cudak, A. Ghosh, Handoff rates for millimeterwave 5g systems, in: 2014 IEEE 79th Vehicular Technology Conference (VTC Spring), IEEE, 2014, pp. 1-5.

[11] A. Barbuzzi, P.H. Perala, G. Boggia, K. Pentikousis, 3gpp radio resource control in practice, IEEE Wirel. Commun. 19 (6) (2012) 76-83.

[12] C. Shen, M. van der Schaar, A learning approach to frequent handover mitigations in 3gpp mobility protocols, in: 2017 IEEE Wireless Communications and Networking Conference (WCNC), IEEE, 2017, pp. 1-6.

[13] M.J. Kochenderfer, T.A. Wheeler, Algorithms for Optimization, Mit Press, 2019.

[14] Y. Koda, K. Yamamoto, T. Nishio, M. Morikura, Reinforcement learning based pre-dictive handover for pedestrian-aware mmwave networks, in: IEEE INFOCOM 2018-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), IEEE, 2018, pp. 692-697.

[15] Y. Koda, K. Nakashima, K. Yamamoto, T. Nishio, M. Morikura, Handover manage-ment for mmwave networks with proactive performance prediction using camera images and deep reinforcement learning, IEEE Trans. Cogn. Commun. Netw. 6(2)(2019) 802-816.

[16] F. Guidolin, I. Pappalardo, A. Zanella, M. Zorzi, Context-aware handover policies in hetnets, IEEE Trans. Wirel. Commun. 15 (3) (2015) 1895-1906.

[17] M. Mezzavilla, S. Goyal, S. Panwar, S. Rangan, M. Zorzi, An mdp model for optimal handover decisions in mmwave cellular networks, in: 2016 European Conference on Networks and Communications (EuCNC), IEEE, 2016, pp. 100-105.

[18] H. Tabrizi, G. Farhadi, J. Cioffi, Dynamic handoff decision in heterogeneous wireless systems: Q-learning approach, in: 2012 IEEE International Conference on Commu-nications (ICC), IEEE, 2012, pp. 3217-3222.

[19] Z. Wang, L. Li, Y. Xu, H. Tian, S. Cui, Handover control in wireless systems via asyn-chronous multiuser deep reinforcement learning, IEEE Int. Things J. 5(6) (2018) 4296-4307.

[20] F. Spitzer, Principles of Random Walk, vol. 34, Springer Science & Business Media, 2013.

[21] S. Zang, W. Bao, P.L. Yeoh, B. Vucetic, Y. Li, Managing vertical handovers in millimeter wave heterogeneous networks, IEEE Trans. Commun. 67 (2) (2018) 1629-1644.

[22] R. Wang, P.V. Klaine, O. Onireti, Y. Sun, M.A. Imran, L. Zhang, Deep learning en-abled beam tracking for non-line of sight millimeter wave communications, IEEE Open J. Commun. Soc. 2 (2021) 1710-1720.

[23] T. Bai, R. Vaze, R.W. Heath, Using random shape theory to model blockage in ran-dom cellular networks, in: 2012 International Conference on Signal Processing and Communications (SPCOM), IEEE, 2012, pp. 1-5.

[24] V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A. Graves, M. Riedmiller, A.K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature 518 (7540) (2015) 529-533.

[25] J. Schulman, P. Moritz, S. Levine, M. Jordan,P. Abbeel, High-dimensional contin-uous control using generalized advantage estimation, arXiv preprint, arXiv :1506. 02438.

[26] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, S. Whiteson,Counterfactual multi-agent policy gradients, in:Proceedings of the AAAI Conference on Artificial Intelli-gence, vol. 32, 2018.

[27] G. Tesauro, Temporal difference learning and td-gammon, Commun. ACM 38 (3)(1995) 58-68.

[28] D.P. Kingma, J. Ba, Adam: a method for stochastic optimization, arXiv preprint, arXiv :1412.6980.

[29] E.U.T.R. Access,Requirements for support of radio resource management (3gpp ts 36.133 version 12.6. 0 release 12), ETSI TS 136 (133) (2015) V11.

[30] S. Wright, J. Nocedal, et al., Numerical optimization, Springer Sci. 35 (67-68)(1999) 7.

[31] I. Kezurer, S.Z. Kovalsky, R. Basri, Y. Lipman, Tight Relaxation of Quadratic Match-ing, Computer Graphics Forum, vol. 34, Wiley Online Library, 2015, pp. 115-128.

[32] A.B. Jambekar, D.I. Steinberg, An implicit enumeration algorithm for the all integer programming problem, Comput. Math. Appl. 4(1) (1978) 15-31.

[33] H. Elshaer, M.N. Kulkarni, F. Boccardi, J.G. Andrews, M. Dohler, Downlink and uplink cell association with traditional macrocells and millimeter wave small cells, IEEE Trans. Wirel. Commun. 15 (9) (2016) 6244-6258.