AoI-aware transmission control in real-time mmwave energy harvesting systems: a risk-sensitive reinforcement learning approach^☆

Abstract

The evolution of enabling technologies in wireless communications has paved the way for supporting novel applications with more demanding QoS requirements, but at the cost of increasing the complexity of optimizing the digital communication chain. In particular, Millimeter Wave (mmWave) communications provide an abundance of bandwidth, and energy harvesting supplies the network with a continual source of energy to facilitate self-sustainability; however, harnessing these technologies is challenging due to the stochastic dynamics of the mmWave channel as well as the random sporadic nature of the harvested energy. In this paper, we aim at the dynamic optimization of update transmissions in mmWave energy harvesting systems in terms of Age of Information (AoI). AoI has recently been introduced to quantify information freshness and is a more stringent QoS metric compared to conventional delay and throughput. However, most prior art has only addressed average-based AoI metrics, which can be insufficient to capture the occurrence of rare but high-impact freshness violation events in time-critical scenarios. We formulate a control problem that aims to minimize the long-term entropic risk measure of AoI samples by configuring the “sense & transmit” of updates. Due to the high complexity of the exponential cost function, we reformulate the problem with an approximated mean-variance risk measure as the new objective. Under unknown system statistics, we propose a two-timescale model-free risk-sensitive reinforcement learning algorithm to compute a control policy that adapts to the trio of channel, energy, and AoI states. We evaluate the efficiency of the proposed scheme through extensive simulations.

Keywords

Age of information / Millimeter wave / Energy harvesting / Risk-sensitive / Model-free reinforcement learning

CRediT authorship contribution statement

Marzieh Sheikhi: Writing - original draft, Visualization, Validation, Software, Methodology, Formal analysis, Conceptualization. Vesal Hakami: Writing - review & editing, Supervision, Project administration, Conceptualization.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References

[1] D. Moltchanov, E. Sopin, V. Begishev, A. Samuylov, Y. Koucheryavy, K. Samouylov,A tutorial on mathematical modeling of 5G/6G millimeter wave and terahertz cellular systems, IEEE Commun. Surv. Tutor. 24 (2) (2022) 1072-1116.

[2] X. Wang, L. Kong, F. Kong, F. Qiu, M. Xia, S. Arnon, G. Chen, Millimeter wave communication: a comprehensive survey, IEEE Commun. Surv. Tutor. 20 (3) (2018) 1616-1653.

[3] D.K. Sah, T. Amgoth, Renewable energy harvesting schemes in wireless sensor net-works: a survey, Inf. Fusion 63 ( 2020) 223-247.

[4] I. Ahmed, M.M. Butt, C. Psomas, A. Mohamed, I. Krikidis, M. Guizani, Survey on energy harvesting wireless communications: challenges and opportunities for radio resource allocation, Comput. Netw. 88 (2015) 234-248.

[5] S. Kaul, R. Yates, M. Gruteser,Real-time status: how often should one update?, in: 2012 Proceedings IEEE INFOCOM, IEEE, 2012, pp. 2731-2735.

[6] R.D. Yates, Y. Sun, D.R. Brown, S.K. Kaul, E. Modiano, S. Ulukus, Age of information: an introduction and survey, IEEE J. Sel. Areas Commun. 39 (5) (2021) 1183-1210.

[7] X. Zhang, Q. Zhu, H.V. Poor,Age-of-information for mURLLC over 6G multimedia wireless networks, in: 2021 55th Annual Conference on Information Sciences and Systems (CISS), IEEE, 2021, pp. 1-6.

[8] I. Kadota, M.S. Rahman, E. Modiano,WiFresh: age-of-information from theory to implementation, in: 2021 International Conference on Computer Communications and Networks (ICCCN), IEEE, 2021, pp. 1-11.

[9] T. Shreedhar, S.K. Kaul, R.D. Yates,ACP: age control protocol for minimizing age of information over the Internet,in:Proceedings of the 24th Annual International Conference on Mobile Computing and Networking, ACM, New York, NY, USA, 2018, pp. 699-701.

[10] J. Pan, A.M. Bedewy, Y. Sun, N.B. Shroff,Minimizing age of information via schedul-ing over heterogeneous channels, in:Proceedings of the Twenty-Second Interna-tional Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mo-bile Networks and Mobile Computing, ACM, New York, NY, USA, 2021, pp. 111-120.

[11] L. Wang, F. Peng, X. Chen, S. Zhou, Optimal update in energy harvesting aided terahertz communications with random blocking, arXiv :2110.07233, 2021.

[12] X. Cao, J. Wang, Y. Cheng, J. Jin, Optimal sleep scheduling for energy-efficient AoI optimization in industrial Internet of things, IEEE Int. Things J. 10 (11) (2023) 9662-9674.

[13] H. Zheng, K. Xiong, P. Fan, Z. Zhong, K. Ben Letaief,Age-based utility maximization for wireless powered networks: a Stackelberg game approach, in: 2019 IEEE Global Communications Conference (GLOBECOM), IEEE, 2019, pp. 1-6.

[14] X. Zhou, I. Koprulu, A. Eryilmaz,Age-optimal multi-channel-scheduling under en-ergy and tolerance constraints, in: IEEE INFOCOM 2023 - IEEE Conference on Com-puter Communications Workshops (INFOCOM WKSHPS), IEEE, 2023, pp. 1-8.

[15] M. Hatami, M. Leinonen, M. Codreanu, AoI minimization in status update control with energy harvesting sensors, IEEE Trans. Commun. 69 (12) (2021) 8335-8351.

[16] M. Hatami, M. Leinonen, Z. Chen, N. Pappas, M. Codreanu, On-demand AoI mini-mization in resource-constrained cache-enabled IoT networks with energy harvest-ing sensors, IEEE Trans. Commun. 70 (11) (2022) 7446-7463.

[17] M. Xie, X. Jia, M. Zhou, Age of information in energy harvesting dual-sensor status update systems with HARQ-MRC, Ad Hoc Netw. 132 (2022) 102866.

[18] W. Jin, J. Sun, K. Chi, S. Zhang, Deep reinforcement learning based scheduling for minimizing age of information in wireless powered sensor networks, Comput. Com-mun. 191 (2022) 1-10.

[19] J. Xu, I. Hou, N. Gautam, Age of information for single buffer systems with vacation server, IEEE Trans. Netw. Sci. Eng. 9 (3) (2022) 1198-1214.

[20] A. Sinha, P.D. Mankar, N. Pappas, H.S. Dhillon,Age of information with on-off ser-vice, in: 2023 IEEE Information Theory Workshop (ITW), IEEE, 2023, pp. 445-450.

[21] N.I. Miridakis, Z. Shi, T.A. Tsiftsis, G. Yang, Extreme age of information for wireless-powered communication systems, IEEE Wirel. Commun. Lett. 11 (4) (2022) 826-830.

[22] H. Wang, X. Xie, J. Yang, Optimizing average age of information in industrial IoT systems under delay constraint, IEEE Trans. Ind. Inform. (2023) 1-10.

[23] A. Mandal, R. Bhattacharjee, A. Sinha,Optimizing age-of-information in adver-sarial environments with channel state information, in: 2022 14th International Conference on COMmunication Systems & NETworkS (COMSNETS), IEEE, 2022, pp. 522-530.

[24] D. Sinha, R. Roy, Optimal scheduling for maximizing information freshness and system performance in industrial cyber-physical systems, Comput. Commun. 169 (2021) 33-47.

[25] M. Pourmohammad Abdollahi, H. Azarhava, A. Haghrah, J. Musevi Niya, On the rate and age of information for non-preemptive systems with prioritized arrivals and deterministic packet deadlines in IoT networks, Ad Hoc Netw. 124 (2022) 102717.

[26] A. Elgabli, H. Khan, M. Krouka, M. Bennis,Reinforcement learning based scheduling algorithm for optimizing age of information in ultra reliable low latency networks, in: 2019 IEEE Symposium on Computers and Communications (ISCC), IEEE, 2019, pp. 1-6.

[27] B. Zhou, W. Saad, M. Bennis, P. Popovski,Risk-aware optimization of age of infor-mation in the Internet of things, in: ICC 2020-2020 IEEE International Conference on Communications (ICC), IEEE, 2020, pp. 1-6.

[28] Q. Zhang, X. Wang, Z. Li, Z. Wei,Design and performance evaluation of joint sensing and communication integrated system for 5G mmWave enabled CAVs, IEEE J. Sel. Top. Signal Process. 15 (6) (2021) 1500-1514.

[29] G. Yao, C. Wang, N.B. Shroff,Age minimization with energy and distortion con-straints, in:In Proceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Com-puting (MobiHoc ’23), ACM, 2023, pp. 101-110.

[30] M.A. Abd-Elmagid, H.S. Dhillon, Age of information in multi-source updating sys-tems powered by energy harvesting, IEEE J. Sel. Areas Inf. Theory 3 (1) (2022) 98-112.

[31] A. Jaiswal, A. Chattopadhyay, A. Varma, Age-of-information minimization via op-portunistic sampling by an energy harvesting source, IEEE Trans. Cogn. Commun. Netw. 10 (6) (2024) 2296-2310.

[32] S. Asadi, M. Sheikhi, V. Hakami,AoI-aware optimization in RIS-assisted mmWave network, in: 2024 10th International Conference on Web Research (ICWR), IEEE, 2024, pp. 175-181.

[33] Z. Guo, Y. Niu, S. Mao, C. Zhang, N. Wang, Z. Zhong, B. Ai, Sum rate maximization under AoI constraints for RIS-assisted mmWave communications, IEEE Trans. Veh. Technol. 73 (4) (2024) 5243-5258.

[34] Y. Khorsandmanesh, M.J. Emadi,Peak age of information analysis for virtual real-ity in terahertz communications, in: 2020 Iran Workshop on Communication and Information Theory (IWCIT), IEEE, 2020, pp. 1-6.

[35] N. Akar, E. Karasan, Is proportional fair scheduling suitable for age-sensitive traffic?, Comput. Netw. 226 (2023) 109668.

[36] Y. Sun, B. Cyr, Sampling for data freshness optimization: non-linear age functions, J. Commun. Netw. 21 (3) (2019) 204-219.

[37] L. Hu, Z. Chen, Y. Dong, Y. Jia, L. Liang, M. Wang, Status update in IoT networks: age-of-information violation probability and optimal update rate, IEEE Int. Things J. 8 (14) (2021) 11329-11344.

[38] C. Chaccour, W. Saad,On the ruin of age of information in augmented reality over wireless terahertz (THz) networks, in: GLOBECOM 2020-2020 IEEE Global Com-munications Conference, IEEE, 2020, pp. 1-6.

[39] E.S. Lohan, M. Koivisto, O. Galinina, S. Andreev, A. Tolli, G. Destino, M. Costa, K. Leppanen, Y. Koucheryavy, M. Valkama, Benefits of positioning-aided communica-tion technology in high-frequency industrial IoT, IEEE Commun. Mag. 56 (12) (2018) 142-148.

[40] J. Huang, H. Gao, S. Wan, Y. Chen, AoI-aware energy control and computation of-floading for industrial IoT, Future Gener. Comput. Syst. 139 (2023) 29-37.

[41] C. Xu, Q. Xu, J. Wang, K. Wu, K. Lu, C. Qiao,AoI-centric task scheduling for au-tonomous driving systems, in: IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, IEEE, 2022, pp. 1019-1028.

[42] Y. Wang, M.P. Chapman, Risk-averse autonomous systems: a brief history and re-cent developments from the perspective of optimal control, Artif. Intell. 311 (2022) 103743.

[43] S. Wang, M. Chen, W. Saad, C. Yin, S. Cui, H.V. Poor,Reinforcement learning for minimizing age of information under realistic physical dynamics, in: GLOBECOM 2020-2020 IEEE Global Communications Conference, IEEE, 2020, pp. 1-6.

[44] R.S. Sutton, A.G. Barto, Reinforcement Learning:An Introduction, The MIT Press, 2018.

[45] M. Sheikhi, V. Hakami,AoI-aware status update control for an energy harvesting source over an uplink mmWave channel, in: 2021 7th International Conference on Signal Processing and Intelligent Systems (ICSPIS), IEEE, 2021, pp. 01-06.

[46] P. Whittle, Risk-sensitive linear/quadratic/Gaussian control, Adv. Appl. Probab. 13 (4) (1981) 764-777.

[47] J.A. Filar, L.C.M. Kallenberg, H.-M. Lee, Variance-penalized Markov decision pro-cesses, Math. Oper. Res. 14 (1) (1989) 147-161.

[48] A. Gosavi, Variance-penalized Markov decision processes: dynamic programming and reinforcement learning techniques, Int. J. Gen. Syst. 43 (6) (2014) 649-669.

[49] A. Gosavi, S.K. Das, S.L. Murray,Beyond exponential utility functions: a variance-adjusted approach for risk-averse reinforcement learning, in: 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), IEEE, 2014, pp. 1-8.

[50] A. Gosavi,Reinforcement learning for model building and variance-penalized con-trol, in:Proceedings of the 2009 Winter Simulation Conference (WSC), IEEE, 2009, pp. 373-379.

[51] S. Bhatnagar, H. Prasad, L. Prashanth, Stochastic Recursive Algorithms for Optimiza-tion: Simultaneous Perturbation Methods, Lecture Notes in Control and Information Sciences, vol. 434, Springer, London, London, 2013.

[52] O. Onireti, A. Imran, M.A. Imran, Coverage, capacity, and energy efficiency analysis in the uplink of mmWave cellular networks, IEEE Trans. Veh. Technol. 67 (5) (2018) 3982-3997.

[53] A. Fayad, T. Cinkler,Energy-efficient joint user and power allocation in 5G mil-limeter wave networks: a genetic algorithm-based approach, IEEE Access 12 (2024) 20019-20030.

[54] Y. Liu, X. Fang, M. Xiao, F. Song, Y. Cui, Q. Xue, C. Tang, Latency optimization for multi-UAV-assisted task offloading in air-ground integrated millimeter-wave net-works, IEEE Trans. Wirel. Commun. 23 (10) (2024) 13359-13376.

[55] B. Lv, R. Wang, Delay-aware two-time-scale scheduling for mmWave systems with mobility and environment knowledge, IEEE Trans. Commun. 72 (7) (2024) 4026-4040.

[56] F. Namjoonia, M. Sheikhi, V. Hakami, Fast reinforcement learning algorithms for joint adaptive source coding and transmission control in IoT devices with renewable energy storage, Neural Comput. Appl. 34 (5) (2022) 3959-3979.

[57] V. Hakami, S. Mostafavi, N.T. Javan, Z. Rashidi, An optimal policy for joint compres-sion and transmission control in delay-constrained energy harvesting IoT devices, Comput. Commun. 160 (1) (2020) 554-566.

[58] S.F.A. Gardazi, J.N. Chattha, M. Uppal, Latency-optimum NOMA with RF energy har-vesting and finite battery capacity, IEEE Commun. Lett. 27 (10) (2023) 2832-2836.

[59] G. Li, M. Zeng, D. Mishra, L. Hao, Z. Ma, O.A. Dobre, Latency minimization for IRS-aided NOMA MEC systems with WPT-enabled IoT devices, IEEE Int. Things J. 10 (14) (2023) 12156-12168.

[60] M. Costa, M. Codreanu, A. Ephremides,Age of information with packet manage-ment, in: 2014 IEEE International Symposium on Information Theory, IEEE, 2014, pp. 1583-1587.

[61] Y. Inoue,The probability distribution of the AoI in queues with infinitely many servers, in: IEEE INFOCOM 2020 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), IEEE, 2020, pp. 297-302.

[62] T.Z. Ornee, Y. Sun, Sampling and remote estimation for the Ornstein-Uhlenbeck pro-cess through queues: age of information and beyond, IEEE/ACM Trans. Netw. 29 (5) (2021) 1962-1975.

[63] Y. Sun, Y. Polyanskiy, E. Uysal, Sampling of the Wiener process for remote estimation over a channel with random delay, IEEE Trans. Inf. Theory 66 (2) (2020) 1118-1135.

[64] T. Zhang, J. Zhou, Z. Chen, Z. Tian, W. Wen, Y. Jia, Information freshness optimiza-tion of multiple status update streams in Internet of things: generation rate control and service rate reservation, Digit. Commun. Netw. 9 (4) (2023) 971-980.

[65] M. Moradian, A. Dadlani,Age of information in scheduled wireless relay networks, in: 2020 IEEE Wireless Communications and Networking Conference (WCNC), IEEE, 2020, pp. 1-6.

[66] A. Fox, F. De Pellegrini, E. Altman,Learning optimal edge processing with offloading and energy harvesting, in:Proceedings of the Int’l ACM Conference on Modeling Analysis and Simulation of Wireless and Mobile Systems, ACM, New York, NY, USA, 2023, pp. 83-92.

[67] Y. Fan, Z. Zhang, H. Li, Message passing based distributed learning for joint resource allocation in millimeter wave heterogeneous networks, IEEE Trans. Wirel. Commun. 18 (5) (2019) 2872-2885.

[68] S. Singh, M.N. Kulkarni, A. Ghosh, J.G. Andrews, Tractable model for rate in self-backhauled millimeter wave cellular networks, IEEE J. Sel. Areas Commun. 33 (10) (2015) 2196-2211.

[69] G. Stamatakis, N. Pappas, A. Traganitis,Control of status updates for energy harvest-ing devices that monitor processes with alarms, in: 2019 IEEE Globecom Workshops (GC Wkshps), IEEE, 2019, pp. 1-6.

[70] E. Gindullina, L. Badia, D. Gunduz, Age-of-information with information source di-versity in an energy harvesting system, IEEE Trans. Green Commun. Netw. 5 (3) (2021) 1529-1540.

[71] W. Huang, W.B. Haskell, Stochastic approximation for risk-aware Markov decision processes, IEEE Trans. Autom. Control 66 (3) (2021) 1314-1320.

[72] A. Liu, V.K.N. Lau, Cache-enabled opportunistic cooperative MIMO for video stream-ing in wireless systems, IEEE Trans. Signal Process. 62 (2) (2014) 390-402.

[73] V. Lau, Y. Cui, Delay-optimal power and subcarrier allocation for OFDMA systems via stochastic approximation, IEEE Trans. Wirel. Commun. 9 (1) (2010) 227-233.

[74] Y. Fei, Z. Yang, Y. Chen, Z. Wang, Exponential Bellman equation and improved regret bounds for risk-sensitive reinforcement learning, in: Advances in Neural Information Processing Systems, 2021, pp. 20436-20446.

[75] E. Noorani, C. Mavridis, J. Baras, Risk-sensitive reinforcement learning with expo-nential criteria, arXiv :2212.09010, 2022.

[76] V.S. Borkar, S.P. Meyn, Risk-sensitive optimal control for Markov decision processes with monotone cost, Math. Oper. Res. 27 (1) (2002) 192-209.

[77] K.M. Smith, M.P. Chapman, On exponential utility and conditional value-at-risk as risk-averse performance criteria, IEEE Trans. Control Syst. Technol. 31 (6) (2023) 2555-2570.

[78] S. Batewela, C.-F. Liu, M. Bennis, H.A. Suraweera, C.S. Hong, Risk-sensitive task fetching and offloading for vehicular edge computing, IEEE Commun. Lett. 24 (3) (2020) 617-621.

[79] C.J.C.H. Watkins, P. Dayan, Q-learning, Mach. Learn. 8 (1992) 279-292.

[80] J. Abounadi, D. Bertsekas, V.S. Borkar, Learning algorithms for Markov decision processes with average cost, SIAM J. Control Optim. 40 (3) (2001) 681-698.

[81] D.P. Bertsekas, Dynamic Programming and Optimal Control, 4th edition, Athena Scientific, 2012.

[82] A. Gosavi,Simulation-Based Optimization, Operations Research/Computer Science Interfaces Series, vol. 55, Springer US, Boston, MA, 2015.

[83] James Davidson, Stochastic Limit Theory an Introduction for Econometricians, Ox-ford Llniversity Press, Oxford, 1994.

[84] 3GPP,Study on channel model for frequencies from 0.5 to 100 GHz, TR 38.901 v 17.0.0, Tech. Rep., 2022.

[85] B. Maham, P. Popovski, Capacity analysis of coordinated multipoint reception for mmWave uplink with blockages, IEEE Trans. Veh. Technol. 69 (12) (2020) 16299-16303.

[86] 3GPP, Summary of Rel-15 Work Items, TR 21.915, v 15.0.0, Tech. Rep., 2019.

[87] A.W. van der Vaart, Asymptotic Statistics, Cambridge University Press, 1998.

[88] J. Wang, J. Hu, G. Min, W. Zhan, A.Y. Zomaya, N. Georgalas, Dependent task offload-ing for edge computing based on deep reinforcement learning, IEEE Trans. Comput. 71 (10) (2022) 2449-2461.

[89] L. Lei, H. Xu, X. Xiong, K. Zheng, W. Xiang, X. Wang, Multiuser resource control with deep reinforcement learning in IoT edge computing, IEEE Int. Things J. 6 (6) (2019) 10119-10133.

[90] Y.L. Hsu, C.-F. Liu, S. Samarakoon, H.-Y. Wei, M. Bennis,Age-optimal power allo-cation in industrial IoT: a risk-sensitive federated learning approach, in: 2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Com-munications (PIMRC), IEEE, 2021, pp. 1323-1328.

[91] M.K. Abdel-Aziz, C.-F. Liu, S. Samarakoon, M. Bennis, W. Saad,Ultra-reliable low-latency vehicular networks: taming the age of information tail, in: 2018 IEEE Global Communications Conference (GLOBECOM), IEEE, 2018, pp. 1-7.

[92] W. Rudin, Real and Complex Analysis, 3rd edition, McGraw-Hill, Inc., New York, NY, USA, 1987.

[93] R.B. Ash, Real Analysis and Probability, Elsevier, 1972.

[94] K.A. Ross, Elementary Analysis, Undergraduate Texts in Mathematics, Springer, New York, New York, NY, 2013.