Security defense strategy algorithm for Internet of Things based on deep reinforcement learning

Abstract

Currently, important privacy data of the Internet of Things (IoT) face extremely high risks of leakage. Attackers persistently engage in continuous attacks on terminal devices to obtain private data of crucial importance. Although significant progress has been made in recent years in deep reinforcement learning defense strategies, most defense methods still face problems such as low defense resource allocation efficiency and insufficient defense coordination capabilities. To solve the above problems, this paper constructs a novel adversarial security scenario and proposes a security game model that integrates defense resource allocation and patrol inspection. Regarding the above game model, this paper designs a deep reinforcement learning algorithm named SDSA to calculate its security defense strategy. SDSA calculates the allocation strategy of the best patrolling strategy that is most suitable for the defender by searching the policy on a multi-dimensional discrete action space, and enables multiple defense agents to cooperate efficiently by training a multi-intelligent Dueling Double Deep Q-Network (D3QN) with prioritized experience replay. Finally, the experimental results show that the SDSA-learned security defense strategy can provide a feasible and effective security protection strategy for defenders against attacks compared to the MADDPG and OptGradFP methods.

Keywords

Internet of Things / Cyber security / Deep reinforcement learning / Game theory

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research is supported by the National Natural Science Foundation of China (62172377, 61872205), the Shandong Provincial Natural Science Foundation (ZR2019MF018), and the Startup Research Foundation for Distinguished Scholars (202112016). The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve the quality of this paper.

References

[1] Y. Wu, H.-N. Dai, H. Wang, Convergence of blockchain and edge computing for secure and scalable IIoT critical infrastructures in industry 4.0, IEEE Internet Things J. 8 (4) (2021) 2300-2317.

[2] A. Ghubaish, T. Salman, M. Zolanvari, D. Unal, A. Al-Ali, R. Jain, Recent advances in the internet-of-medical-things (IoMT) systems security, IEEE Internet Things J. 8 (11) (2021) 8707-8718.

[3] Z. Cai, X. Zheng, J. Wang, Z. He, Private data trading towards range counting queries in internet of things, IEEE Trans. Mob. Comput. 22 (8) (2023) 4881-4897.

[4] A.E. Omolara, A. Alabdulatif, O.I. Abiodun, M. Alawida, A. Alabdulatif, H. Arshad, et al., The internet of things security: A survey encompassing unexplored areas and new insights, Comput. Secur. 112 (2022) 102494.

[5] H. HaddadPajouh, A. Dehghantanha, R.M. Parizi, M. Aledhari, H. Karimipour, A survey on internet of things security: Requirements, challenges, and solutions, Internet Things 14 (2021) 100129.

[6] Z. Cai, Z. He, X. Guan, Y. Li, Collective data-sanitization for preventing sensitive information inference attacks in social networks, IEEE Trans. Dependable Secure Comput. 15 (4) (2018) 577-590.

[7] A. Uprety, D.B. Rawat, Reinforcement learning for iot security: A comprehensive survey, IEEE Internet Things J. 8 (11) (2021) 8693-8706.

[8] T.T. Nguyen, V.J. Reddi, Deep reinforcement learning for cyber security, IEEE Trans. Neural Netw. Learn. Syst. (2021) 1-17.

[9] T. Chen, J. Liu, Y. Xiang, W. Niu, E. Tong, Z. Han, Adversarial attack and defense in reinforcement learning-from AI security view, Cybersecurity 2 (1) (2019) 1-22.

[10] Y. Wu, Z. Wang, Y. Ma, V.C. Leung, Deep reinforcement learning for blockchain in industrial IoT: A survey, Comput. Netw. 191 (2021) 108004.

[11] Z. Cai, X. Zheng, A private and efficient mechanism for data uploading in smart cyber-physical systems, IEEE Trans. Netw. Sci. Eng. 7 (2) (2020) 766-775.

[12] A. Chowdhary, S. Sengupta, D. Huang, S. Kambhampati, Markov game modeling of moving target defense for strategic detection of threats in cloud networks, 2019, arXiv preprint.

[13] Z. Ni, S. Paul, A multistage game in smart grid security: A reinforcement learning solution, IEEE Trans. Neural Netw. Learn. Syst. 30 (9) (2019) 2684-2695.

[14] K. Hammar, R. Stadler, Finding effective security strategies through reinforcement learning and self-play, in:Proceedings of the 16th International Conference on Network and Service Management, 2020, pp. 1-9, https://doi.org/10.23919/CNSM50824.2020.9269092.

[15] K. Hammar, R. Stadler, Learning security strategies through game play and optimal stopping, 2022, arXiv preprint.

[16] L. Huang, M. Fu, H. Qu, S. Wang, S. Hu, A deep reinforcement learning-based method applied for solving multi-agent defense and attack problems, Expert Syst. Appl. 176 (2021) 114896.

[17] S. Sengupta, S. Kambhampati, Multi-agent reinforcement learning in bayesian stackelberg markov games for adaptive moving target defense, 2020, arXiv preprint.

[18] C. Huang, G. Chen, K.-K. Wong, Multi-agent reinforcement learning-based buffer-aided relay selection in IRS-assisted secure cooperative networks, IEEE Trans. Inf. Forensics Secur. 16 (2021) 4101-4112.

[19] M. Xu, J. Peng, B. Gupta, J. Kang, Z. Xiong, Z. Li, A.A. Abd El-Latif, Multiagent federated reinforcement learning for secure incentive mechanism in intelligent cyber-physical systems, IEEE Internet Things J. 9 (22) (2021) 22095-22108.

[20] Z. Cai, Z. He, Trading private range counting over big IoT data, in: 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), 2019, pp. 144-153, http://dx.doi.org/10.1109/ICDCS.2019.00023.

[21] T.T. Nguyen, N.D. Nguyen, S. Nahavandi, Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications, IEEE Trans. Cybern. 50 (9) (2020) 3826-3839.

[22] A. Wong, T. Bäck, A.V. Kononova, A. Plaat, Deep multiagent reinforcement learning: challenges and directions, Artif. Intell. Rev. 56 (2023) 5023-5056.

[23] X. Zheng, Z. Cai, Privacy-preserved data sharing towards multiple parties in industrial IoTs, IEEE J. Sel. Areas Commun. 38 (5) (2020) 968-979.

[24] M. Prajapat, K. Azizzadenesheli, A. Liniger, Y. Yue, A. Anandkumar, Competitive policy optimization, in:Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence, Vol. 161, PMLR, 2021, pp. 64-74, URL: https://proceedings.mlr.press/v161/prajapat21a.html.

[25] O. Vinyals, I. Babuschkin, W.M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D.H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature 575 (7782) (2019) 350-354.

[26] A.P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z.D. Guo, C. Blundell, Agent57: Outperforming the atari human benchmark,in:Proceedings of the 37th International Conference on Machine Learning, Vol. 119, PMLR, 2020, pp. 507-517, URL: https://proceedings.mlr.press/v119/badia20a.html.

[27] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, N. Freitas, Dueling network architectures for deep reinforcement learning, in:Proceedings of the 33nd International Conference on Machine Learning, Vol. 48, PMLR, 2016, pp. 1995-2003, URL: https://proceedings.mlr.press/v48/wangf16.html.

[28] Y. Chandak, G. Theocharous, J. Kostas, S. Jordan, P. Thomas, Learning action representations for reinforcement learning, in:Proceedings of the 36th International Conference on Machine Learning, Vol. 97, PMLR, 2019, pp. 941-950, URL: https://proceedings.mlr.press/v97/chandak19a.html.

[29] R. Lowe, Y.I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, I. Mordatch, Multiagent actor-critic for mixed cooperative-competitive environments, Adv. Neural Inf. Process. Syst. 30 (2017) 6379-6390.

[30] N. Kamra, U. Gupta, F. Fang, Y. Liu, M. Tambe, Policy learning for continuous space security games using neural networks, in:Proceedings of the 32th AAAI Conference on Artificial Intelligence, Vol. 32, 2018, pp. 1103-1112, URL: https://ojs.aaai.org/index.php/AAAI/article/view/11446. https://doi.org/10.1609/aaai.v32i1.11446.

[31] M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, T. Graepel, A unified game-theoretic approach to multiagent reinforcement learning, Adv. Neural Inf. Process. Syst. 30 (2017) 4190-4203.