Deep reinforcement learning-based spectrum resource allocation for the web of healthcare things with massive integrating wearable gadgets☆

Abstract

With the development of the future Web of Healthcare Things (WoHT), there will be a trend of densely deploying medical sensors with massive simultaneous online communication requirements. The dense deployment and simultaneous online communication of massive medical sensors will inevitably generate overlapping interference. This will be extremely challenging to support data transmission at the medical-grade quality of service level. To handle the challenge, this paper proposes a hypergraph interference coordination-aided resource allocation based on the Deep Reinforcement Learning (DRL) method. Specifically, we build a novel hypergraph interference model for the considered WoHT by analyzing the impact of the overlapping interference. Due to the high complexity of directly solving the hypergraph interference model, the original resource allocation problem is converted into a sequential decision-making problem through the Markov Decision Process (MDP) modeling method. Then, a policy and value-based resource allocation algorithm is proposed to solve this problem under simultaneous online communication and dense deployment. In addition, to enhance the exploration ability of the optimal allocation strategy for the agent, we propose a resource allocation algorithm with an asynchronous parallel architecture. Simulation results verify that the proposed algorithms can achieve higher network throughput than the existing algorithms in the considered WoHT scenario.

Keywords

Web of healthcare things / Hypergraph / Interference coordination / Deep reinforcement learning

CRediT authorship contribution statement

Jie Huang: Writing - original draft. Cheng Yang: Writing - original draft. Fan Yang: Writing - original draft. Shilong Zhang: Writing - original draft. Amr Tolba: Resources. Alireza Jolfaei: Resources. Keping Yu: Resources.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62301094, in part by the Researchers Supporting Project Number (RSPD2024R681), King Saud University, Riyadh, Saudi Arabia, in part by the Science and Technology Research Program of the Chongqing Education Commission of China under Grants KJQN202201157 and KJQN202301135.

References

[1] A. Entezami, H. Sarmadi, B. Behkamal, Long-term health monitoring of concrete and steel bridges under large and missing data by unsupervised meta learning, Eng. Struct. 279 (2023) 115616.

[2] J. Huang, T. Yu, F. Yang, S. Zhang, W. Jiang, D. Niyato, Aoi-aware resource allocation with interference avoidance for ultra-dense industrial Internet of things networks, IEEE Int. Things J., https://doi.org/10.1109/JIOT.2024.3403849.

[3] A.M. Seid, A. Erbad, H.N. Abishu, A. Albaseer, M. Abdallah, M. Guizani, Multiagent federated reinforcement learning for resource allocation in uav-enabled Internet of medical things networks, IEEE Int. Things J. 10 (22) (2023) 19695-19711.

[4] S. Vikas Shinde, S.S. Sonavane,An electronic tattoo based wireless body area net-work (et-wban) to conform ieee 802.15.6: a review and proposal, in: 2018 Inter-national Conference on Computing, Power and Communication Technologies (GU-CON), IEEE, 2018, pp. 269-274.

[5] M. Dubey, R. Mishra, 5g in healthcare: revolutionary use-cases and qos provisioning powered by network slicing,in: Intelligent Systems and Smart Infrastructure, CRC Press, 2023, pp. 96-106.

[6] J. Huang, S. Zhang, F. Yang, T. Yu, L.V.N. Prasad, M. Guduri, K. Yu, Hypergraph-based interference avoidance resource management in customer-centric communi-cation for intelligent cyber-physical transportation systems, IEEE Trans. Consum. Electron. 70 (1) (2024) 1775-1786.

[7] S. Aminizadeh, A. Heidari, S. Toumaj, M. Darbandi, N.J. Navimipour, M. Rezaei, S. Talebi, P. Azad, M. Unal, The applications of machine learning techniques in medical data processing based on distributed computing and the Internet of things, Comput. Methods Programs Biomed. 241 (2023) 107745.

[8] J. Huang, F. Yang, C. Chakraborty, Z. Guo, H. Zhang, L. Zhen, K. Yu,Opportunis-tic capacity based resource allocation for 6g wireless systems with network slicing, Future Gener. Comput. Syst. 140 (2023) 390-401.

[9] F. Yang, C. Yang, J. Huang, K. Yu, S. Garg, M. Alrashoud,Hypergraph-based resource-efficient collaborative reinforcement learning for b5g massive iot, IEEE Open J. Commun. Soc. 4 (2023) 2439-2450.

[10] J. Huang, C. Yang, S. Zhang, F. Yang, O. Alfarraj, V. Frascolla, S. Mumtaz, K. Yu,Reinforcement learning based resource management for 6g-enabled miot with hy-pergraph interference model, IEEE Trans. Commun. 72 (7) (2024) 4179-4192.

[11] F. Yang, S. Zhang, C. Liu, J. Huang, T. Yu, K. Yu, M. Guizani, A hierarchical network management strategy for distributed ciiot with imperfect csi, IEEE Int. Things J. 11 (8) (2024) 13509-13523.

[12] L. Liang, J. Kim, S.C. Jha, K. Sivanesan, G.Y. Li, Spectrum and power allocation for vehicular communications with delayed csi feedback, IEEE Wirel. Commun. Lett. 6 (4) (2017) 458-461.

[13] L. Zhang, Y.-C. Liang, Joint spectrum sensing and packet error rate optimization in cognitive iot, IEEE Int. Things J. 6 (5) (2019) 7816-7827.

[14] A. Shahini, A. Kiani, N. Ansari, Energy efficient resource allocation in eh-enabled cr networks for iot, IEEE Int. Things J. 6 (2) (2019) 3186-3193.

[15] G. Ramkumar, J. Seetha, R. Priyadarshini, M. Gopila, G. Saranya, Iot-based patient monitoring system for predicting heart disease using deep learning, Measurement 218 (2023) 113235.

[16] Z. Shi, X. Xie, H. Lu, H. Yang, M. Kadoch, M. Cheriet, Deep-reinforcement-learning-based spectrum resource management for industrial Internet of things, IEEE Int. Things J. 8 (5) (2021) 3476-3489.

[17] H. Liang, X. Zhang, X. Hong, Z. Zhang, M. Li, G. Hu, F. Hou, Reinforcement learning enabled dynamic resource allocation in the Internet of vehicles, IEEE Trans. Ind. Inform. 17 (7) (2021) 4957-4967.

[18] J. Huang, Y. Yang, Z. Gao, D. He, D.W.K. Ng,Dynamic spectrum access for d2d-enabled Internet of things: a deep reinforcement learning approach, IEEE Int. Things J. 9 (18) (2022) 17793-17807.

[19] N.-P. Tran, T.P. Truong, Q.T. Do, N.-N. Dao, S. Cho, Joint wireless resource allo-cation and bitrate adaptation for qoe improvement in irs-aided rsma-enabled iomt streaming systems, Int. Things 25 (2024) 101145.

[20] W. Lei, Y. Ye, M. Xiao, Deep reinforcement learning-based spectrum allocation in integrated access and backhaul networks, IEEE Trans. Cogn. Commun. Netw. 6 (3) (2020) 970-979.

[21] C. Pan, H. Zhu, N.J. Gomes, J. Wang, Joint user selection and energy minimization for ultra-dense multi-channel c-ran with incomplete csi, IEEE J. Sel. Areas Commun. 35 (8) (2017) 1809-1824.

[22] C. Huang, G. Chen, Y. Gong, P. Xu, Z. Han, J.A. Chambers, Buffer-aided relay selec-tion for cooperative hybrid noma/oma networks with asynchronous deep reinforce-ment learning, IEEE J. Sel. Areas Commun. 39 (8) (2021) 2514-2525.

[23] Y. Azimi, S. Yousefi, H. Kalbkhani, T. Kunz,Energy-efficient deep reinforcement learning assisted resource allocation for 5g-ran slicing, IEEE Trans. Veh. Technol. 71 (1) (2022) 856-871.

[24] Z. Chang, J. Gong, T. Ristaniemi, Z. Niu, Energy-efficient resource allocation and user scheduling for collaborative mobile clouds with hybrid receivers, IEEE Trans. Veh. Technol. 65 (12) (2016) 9834-9846.

[25] Q. Hu, Y. Liu, Y. Cai, G. Yu, Z. Ding, Joint deep reinforcement learning and unfolding: beam selection and precoding for mmwave multiuser mimo with lens arrays, IEEE J. Sel. Areas Commun. 39 (8) (2021) 2289-2304.

[26] F. Tang, H. Hofner, N. Kato, K. Kaneko, Y. Yamashita, M. Hangai, A deep rein-forcement learning-based dynamic traffic offloading in space-air-ground integrated networks (sagin), IEEE J. Sel. Areas Commun. 40 (1) (2022) 276-289.

[27] V. Mnih, A.P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, K. Kavukcuoglu,Asynchronous methods for deep reinforcement learning, in:Interna-tional Conference on Machine Learning, PMLR, 2016, pp. 1928-1937.

[28] Y.M. Park, S.S. Hassan, Y.K. Tun, Z. Han, C.S. Hong, Joint trajectory and resource op-timization of mec-assisted uavs in sub-thz networks: a resources-based multi-agent proximal policy optimization drl with attention mechanism, IEEE Trans. Veh. Tech-nol. 73 (2) (2024) 2003-2016.