Deep reinforcement learning based latency-energy minimization in smart healthcare network

Abstract

Significant breakthroughs in the Internet of Things (IoT) and 5G technologies have driven several smart healthcare activities, leading to a flood of computationally intensive applications in smart healthcare networks. Mobile Edge Computing (MEC) is considered as an efficient solution to provide powerful computing capabilities to latency or energy sensitive nodes. The low-latency and high-reliability requirements of healthcare application services can be met through optimal offloading and resource allocation for the computational tasks of the nodes. In this study, we established a system model consisting of two types of nodes by considering nondivisible and trade-off computational tasks between latency and energy consumption. To minimize processing cost of the system tasks, a Mixed-Integer Nonlinear Programming (MINLP) task offloading problem is proposed. Furthermore, this problem is decomposed into task offloading decisions and resource allocation problems. The resource allocation problem is solved using traditional optimization algorithms, and the offloading decision problem is solved using a deep reinforcement learning algorithm. We propose an Online Offloading based on the Deep Reinforcement Learning (OO-DRL) algorithm with parallel deep neural networks and a weight-sensitive experience replay mechanism. Simulation results show that, compared with several existing methods, our proposed algorithm can perform real-time task offloading in a smart healthcare network in dynamically varying environments and reduce the system task processing cost.

Keywords

Smart healthcare network / Mobile edge computing / Resource allocation / Computation offloading / Deep reinforcement learning

CRediT authorship contribution statement

Xin Su: Methodology, Investigation, Funding acquisition, Conceptualization. Xin Fang: Writing - review & editing, Writing - original draft, Validation, Resources, Investigation, Conceptualization. Zhen Cheng: Validation, Resources. Ziyang Gong: Data curation. Chang Choi: Supervision, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 62371181, and in part by the Changzhou Science and Technology International Cooperation Program under Grant CZ20230029. This work was also supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government. (MSIT) (2021R1A2B5B02087169). It was also supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (RS-2023-00259004) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation)

References

[1] A. Gatouillat, Y. Badr, B. Massot, E. Sejdi´c, Internet of medical things: a review of recent contributions dealing with cyber-physical systems in medicine, IEEE Int. Things J. 5 (5) (2018) 3810-3822.

[2] M. Chen, J. Yang, J. Zhou, Y. Hao, J. Zhang, C.-H. Youn, 5g-smart diabetes: toward personalized diabetes diagnosis with healthcare big data clouds, IEEE Commun. Mag. 56 (4) (2018) 16-23.

[3] Z. Ning, P. Dong, X. Wang, X. Hu, L. Guo, B. Hu, Y. Guo, T. Qiu, R.Y. Kwok,Mo-bile edge computing enabled 5g health monitoring for Internet of medical things: a decentralized game theoretic approach, IEEE J. Sel. Areas Commun. 39 (2) (2021) 463-478.

[4] M. Asif-Ur-Rahman, F. Afsana, M. Mahmud, M.S. Kaiser, M.R. Ahmed, O. Kaiwartya, A. James-Taylor, Toward a heterogeneous mist, fog, and cloud-based framework for the Internet of healthcare things, IEEE Int. Things J. 6 (3) (2019) 4049-4062.

[5] Y.-T. Tsai, Z.Y. Lin,A survey on edge computing in bioinformatics and health infor-matics, in: 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), IEEE, 2020, pp. 2203-2208.

[6] X. Lin, J. Wu, A.K. Bashir, W. Yang, A. Singh, A.A. AlZubi,Fairhealth: long-term proportional fairness-driven 5g edge healthcare in Internet of medical things, IEEE Trans. Ind. Inform. 18 (12) (2022) 8905-8915.

[7] L. Zhang, B. Cao, Y. Li, M. Peng, G. Feng, A multi-stage stochastic programming-based offloading policy for fog enabled iot-ehealth, IEEE J. Sel. Areas Commun. 39 (2) (2021) 411-425.

[8] A.P. Plageras, K.E. Psannis, Iot-based health and emotion care system, ICT Expr. 9 (1) (2023) 112-115.

[9] R. Yadav, W. Zhang, I.A. Elgendy, G. Dong, M. Shafiq, A.A. Laghari, S. Prakash, Smart healthcare: Rl-based task offloading scheme for edge-enable sensor networks, IEEE Sens. J. 21 (22) (2021) 24910-24918.

[10] H. Yang, Z. Wei, Z. Feng, X. Chen, Y. Li, P. Zhang, Intelligent computation offload-ing for mec-based cooperative vehicle infrastructure system: a deep reinforcement learning approach, IEEE Trans. Veh. Technol. 71 (7) (2022) 7665-7679.

[11] H. Li, S. Wu, J. Jiao, X.-H. Lin, N. Zhang, Q. Zhang, Energy-efficient task offload-ing of edge-aided maritime uav systems, IEEE Trans. Veh. Technol. 72 (1) (2023) 1116-1126.

[12] M. Tang, V.W. Wong, Deep reinforcement learning for task offloading in mobile edge computing systems, IEEE Trans. Mob. Comput. 21 (6) (2022) 1985-1997.

[13] X. Su, Z. Wang, Y. Wang, S. Zhou, Multi-access edge computing offloading in mar-itime monitoring sensor networks, Chin. J. Int. Things 5 (1) (2021) 36-52.

[14] S. Liu, Y. Yu, L. Guo, P.L. Yeoh, B. Vucetic, Y. Li, Adaptive delay-energy balanced partial offloading strategy in mobile edge computing networks, Digit. Commun. Netw. 9 (6) (2023) 1310-1318.

[15] Y. Mao, C. You, J. Zhang, K. Huang, K.B. Letaief, A survey on mobile edge com-puting: the communication perspective, IEEE Commun. Surv. Tutor. 19 (4) (2017) 2322-2358.

[16] H. Zhou, W. Xu, J. Chen, W. Wang,Evolutionary v2x technologies toward the Inter-net of vehicles: challenges and opportunities, Proc. IEEE 108 (2) (2020) 308-323.

[17] R. Cong, Z. Zhao, G. Min, C. Feng, Y. Jiang,Edgego: a mobile resource-sharing framework for 6g edge computing in massive iot systems, IEEE Int. Things J. 9 (16) (2022) 14521-14529.

[18] K. Sadatdiynov, L. Cui, L. Zhang, J.Z. Huang, S. Salloum, M.S. Mahmud, A review of optimization methods for computation offloading in edge computing networks, Digit. Commun. Netw. 9 (2) (2023) 450-461.

[19] M.P.J. Mahenge, C. Li, C.A. Sanga, Energy-efficient task offloading strategy in mo-bile edge computing for resource-intensive mobile applications, Digit. Commun. Netw. 8 (6) (2022) 1048-1058.

[20] S. Anbalagan, D. Kumar, G. Raja, A. Balaji,Sdn assisted Stackelberg game model for lte-wifi offloading in 5g networks, Digit. Commun. Netw. 5 (4) (2019) 268-275.

[21] W. Feng, N. Zhang, S. Li, S. Lin, R. Ning, S. Yang, Y. Gao, Latency minimization of reverse offloading in vehicular edge computing, IEEE Trans. Veh. Technol. 71 (5) (2022) 5343-5357.

[22] J. Du, F.R. Yu, X. Chu, J. Feng, G. Lu, Computation offloading and resource alloca-tion in vehicular networks based on dual-side cost minimization, IEEE Trans. Veh. Technol. 68 (2) (2019) 1079-1092.

[23] X. Hu, L. Wang, K.-K. Wong, M. Tao, Y. Zhang, Z. Zheng, Edge and central cloud computing: a perfect pairing for high energy efficiency and low-latency, IEEE Trans. Wirel. Commun. 19 (2) (2020) 1070-1083.

[24] M. Mukherjee, V. Kumar, D. Maity, R. Matam, C.X. Mavromoustakis, Q. Zhang, G. Mastorakis,Delay-sensitive and priority-aware task offloading for edge computing-assisted healthcare services, in: GLOBECOM 2020-2020 IEEE Global Communica-tions Conference, IEEE, 2020, pp. 1-5.

[25] H. Materwala, L. Ismail,Energy-aware edge-cloud computation offloading for smart connected health, in: 2021 8th International Conference on Future Internet of Things and Cloud (FiCloud), IEEE, 2021, pp. 144-150.

[26] X. Ma, J. Zhao, Q. Li, Y. Gong,Reinforcement learning based task offloading and take-back in vehicle platoon networks, in: 2019 IEEE International Conference on Communications Workshops (ICC Workshops), IEEE, 2019, pp. 1-6.

[27] S. Xu, Q. Liu, B. Gong, F. Qi, S. Guo, X. Qiu, C. Yang, Rjcc: reinforcement-learning-based joint communicational-and-computational resource allocation mechanism for smart city iot, IEEE Int. Things J. 7 (9) (2020) 8059-8076.

[28] T.Q. Dinh, Q.D. La, T.Q. Quek, H. Shin, Learning for computation offloading in mobile edge computing, IEEE Trans. Commun. 66 (12) (2018) 6353-6367.

[29] F. Jiang, K. Wang, L. Dong, C. Pan, W. Xu, K. Yang, Deep-learning-based joint re-source scheduling algorithms for hybrid mec networks, IEEE Int. Things J. 7 (7) (2020) 6252-6265.

[30] L. Liang, H. Ye, G.Y. Li, Spectrum sharing in vehicular networks based on multi-agent reinforcement learning, IEEE J. Sel. Areas Commun. 37 (10) (2019) 2282-2292.

[31] X. Yuan, Z. Zhang, C. Feng, Y. Cui, S. Garg, G. Kaddoum, K. Yu, A dqn-based frame aggregation and task offloading approach for edge-enabled iomt, IEEE Trans. Netw. Sci. Eng. 10 (3) (2023) 1339-1351.

[32] L. Liu, Y. Zhou, V. Garcia, L. Tian, J. Shi, Load aware joint comp clustering and inter-cell resource scheduling in heterogeneous ultra dense cellular networks, IEEE Trans. Veh. Technol. 67 (3) (2018) 2741-2755.

[33] A. Meng, G. Wei, Y. Zhao, X. Gao, Z. Yang, Green resource allocation for mobile edge computing, Digit. Commun. Netw. 9 (5) (2023) 1190-1199.

[34] J. Yang, Y. Sun, Y. Lei, Z. Zhang, Y. Li, Y. Bao, Z. Lv,Reinforcement learning based edge computing in b5g, Digit. Commun. Netw. 10 (1) (2024) 1-6.

[35] N. Chen, S. Zhang, Z. Qian, J. Wu, S. Lu,When learning joins edge: real-time propor-tional computation offloading via deep reinforcement learning, in: 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS), IEEE, 2019, pp. 414-421.

[36] X. Su, H. Xue, Y. Zhou, J. Zhu, Research on computing offloading method for mar-itime observation monitoring sensor network, J. Commun. 42 (5) (2021) 149-163.

[37] S. Dong, Y. Xia, J. Kamruzzaman, Quantum particle swarm optimization for task offloading in mobile edge computing, IEEE Trans. Ind. Inform. 19 (8) (2023) 9113-9122.

[38] Y. Chen, J. Zhao, Y. Wu, J. Huang, X. Shen, Qoe-aware decentralized task offloading and resource allocation for end-edge-cloud systems: a game-theoretical approach, IEEE Trans. Mob. Comput. 23 (1) (2024) 769-784.