Reinforcement learning-enabled swarm intelligence method for computation task offloading in Internet-of-Things blockchain^☆

Abstract

Blockchain technology, based on decentralized data storage and distributed consensus design, has become a promising solution to address data security risks and provide privacy protection in the Internet-of-Things (IoT) due to its tamper-proof and non-repudiation features. Although blockchain typically does not require the endorsement of third-party trust organizations, it mostly needs to perform necessary mathematical calculations to prevent malicious attacks, which results in stricter requirements for computation resources on the participating devices. By offloading the computation tasks required to support blockchain consensus to edge service nodes or the cloud, while providing data privacy protection for IoT applications, it can effectively address the limitations of computation and energy resources in IoT devices. However, how to make reasonable offloading decisions for IoT devices remains an open issue. Due to the excellent self-learning ability of Reinforcement Learning (RL), this paper proposes a RL enabled Swarm Intelligence Optimization Algorithm (RLSIOA) that aims to improve the quality of initial solutions and achieve efficient optimization of computation task offloading decisions. The algorithm considers various factors that may affect the revenue obtained by IoT devices executing consensus algorithms (e.g., Proof-of-Work), it optimizes the proportion of sub-tasks to be offloaded and the scale of computing resources to be rented from the edge and cloud to maximize the revenue of devices. Experimental results show that RLSIOA can obtain higher-quality offloading decision-making schemes at lower latency costs compared to representative benchmark algorithms.

Keywords

Blockchain / Task offloading / Swarm intelligence / Reinforcement learning

CRediT authorship contribution statement

Zhuo Chen: Writing - review & editing, Supervision, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization. Jiahuan Yi: Writing - original draft, Visualization, Validation, Software, Methodology, Investigation. Yang Zhou: Writing - review & editing, Resources, Project administration, Funding acquisition. Wei Luo: Funding acquisition.

Declaration of Competing Interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled.

Authors: Zhuo Chen, Jiahuan Yi, Yang Zhou, Wei Luo

Acknowledgements

This work has been jointly supported by the Project of Science and Technology Research Program of Chongqing Education Commission of China (No. KJZD-K202401105), High-Quality Development Action Plan for Graduate Education at Chongqing University of Technology (No. gzljg2023308, No. gzljd2024204), the Graduate Innovation Program of Chongqing University of Technology (No. gzlcx20233197) and Yunnan Provincial Key R&D Program (202203AA080006).

References

[1] S. Khanam, I.B. Ahmedy, M.Y. Idna Idris, M.H. Jaward, A.Q. Bin Md Sabri, A sur-vey of security challenges, attacks taxonomy and advanced countermeasures in the Internet of Things, IEEE Access 8 (2020) 219709-219743.

[2] V. Hassija, V. Chamola, V. Saxena, D. Jain, P. Goyal, B. Sikdar, A survey on iot security: application areas, security threats, and solution architectures, IEEE Access 7 (2019) 82721-82743.

[3] F.A. Alaba, M. Othman, I.A.T. Hashem, F. Alotaibi, Internet of things security: a survey, J. Netw. Comput. Appl. 88 (2017) 10-28.

[4] J. Feng, L.T. Yang, Q. Zhu, K.-K.R. Choo, Privacy-preserving tensor decomposition over encrypted data in a federated cloud environment, IEEE Trans. Dependable Se-cure Comput. 17 (4) (2020) 857-868.

[5] J. Feng, L.T. Yang, B. Ren, D. Zou, M. Dong, S. Zhang, Tensor recurrent neural net-work with differential privacy, IEEE Trans. Comput. 73 (3) (2024) 683-693.

[6] P. Zhang, X. Cheng, S. Su, N. Wang, Task allocation under geo-indistinguishability via group-based noise addition, IEEE Trans. Big Data 9 (3) (2023) 860-877.

[7] P. Zhang, X. Cheng, S. Su, N. Wang, Effective truth discovery under local differential privacy by leveraging noise-aware probabilistic estimation and fusion, Knowl.-Based Syst. 261 (2023) 110213.

[8] A. Dorri, S.S. Kanhere, R. Jurdak, P. Gauravaram,Blockchain for iot security and privacy: the case study of a smart home,in:Proceedings of the 2017 IEEE Interna-tional Conference on Pervasive Computing and Communications Workshops, IEEE, 2017, pp. 618-623.

[9] M.A. Khan, K. Salah, Iot security: review, blockchain solutions, and open challenges, Future Gener. Comput. Syst. 82 (2018) 395-411.

[10] T. Li, H. Wang, D. He, J. Yu, Blockchain-based privacy-preserving and rewarding private data sharing for iot, IEEE Int. Things J. 9 (16) (2022) 15138-15149.

[11] S. Nakamoto, Bitcoin: a peer-to-peer electronic cash system, Decentralized business review, 2008.

[12] C.T. Nguyen, D.T. Hoang, D.N. Nguyen, D. Niyato, H.T. Nguyen, E. Dutkiewicz, Proof-of-stake consensus mechanisms for future blockchain networks: fundamentals, applications and opportunities, IEEE Access 7 (2019) 85727-85745.

[13] S.H. Alsamhi, A.V. Shvetsov, S.V. Shvetsova, A. Hawbani, M. Guizani, M.A. Alhar-tomi, O. Ma, Blockchain-empowered security and energy efficiency of drone swarm consensus for environment exploration, IEEE Trans. Green Commun. Netw. 7 (1) (2023) 328-338.

[14] S. Huang, H. Huang, G. Gao, Y.-E. Sun, Y. Du, J. Wu, Edge resource pricing and scheduling for blockchain: a Stackelberg game approach, IEEE Trans. Serv. Comput. 16 (2) (2023) 1093-1106.

[15] A.A. Mamun, S. Azam, C. Gritti, Blockchain-based electronic health records manage-ment: a comprehensive review and future research direction, IEEE Access 10 (2022) 5768-5789.

[16] Z. Xiong, Y. Zhang, D. Niyato, P. Wang, Z. Han, When mobile blockchain meets edge computing, IEEE Commun. Mag. 56 (8) (2018) 33-39.

[17] C. Qiu, H. Yao, C. Jiang, S. Guo, F. Xu, Cloud computing assisted blockchain-enabled Internet of Things, IEEE Trans. Cloud Comput. 10 (1) (2022) 247-257.

[18] Y. Wei, Z. An, S. Leng, K. Yang, Evolved pow: integrating the matrix computa-tion in machine learning into blockchain mining, IEEE Int. Things J. 10 (8) (2023) 6689-6702.

[19] Y. Wan, Y. Zhong, A. Ma, L. Zhang,An accurate UAV 3-d path planning method for disaster emergency response based on an improved multiobjective swarm intel-ligence algorithm, IEEE Trans. Cybern. 53 (4) (2023) 2658-2671.

[20] J. Zhou, Y. Shen, L. Li, C. Zhuo, M. Chen, Swarm intelligence-based task schedul-ing for enhancing security for iot devices, IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 42 (6) (2023) 1756-1769.

[21] O.N. Oyelade, A.E.-S. Ezugwu, T.I.A. Mohamed, L. Abualigah, Ebola optimization search algorithm: a new nature-inspired metaheuristic optimization algorithm, IEEE Access 10 (2022) 16150-16177.

[22] I. Attiya, M.A. Elaziz, L. Abualigah, T.N. Nguyen, A.A.A. El-Latif, An improved hybrid swarm intelligence for scheduling iot application tasks in the cloud, IEEE Trans. Ind. Inform. 18 (9) (2022) 6264-6272.

[23] M. Karimi-Mamaghan, M. Mohammadi, P. Meyer, A.M. Karimi-Mamaghan, E.-G. Talbi, Machine learning at the service of meta-heuristics for solving combinatorial optimization problems: a state-of-the-art, Eur. J. Oper. Res. 296 (2) (2022) 393-422.

[24] Z. Chen, Z. Yu, Intelligent offloading in blockchain-based mobile crowdsensing using deep reinforcement learning, IEEE Commun. Mag. 61 (6) (2023) 118-123.

[25] M. Li, P. Pei, F.R. Yu, P. Si, Y. Li, E. Sun, Y. Zhang, Cloud-edge collaborative resource allocation for blockchain-enabled Internet of things: a collective reinforcement learn-ing approach, IEEE Int. Things J. 9 (22) (2022) 23115-23129.

[26] H. Wu, K. Wolter, P. Jiao, Y. Deng, Y. Zhao, M. Xu, Eedto: an energy-efficient dy-namic task offloading algorithm for blockchain-enabled iot-edge-cloud orchestrated computing, IEEE Int. Things J. 8 (4) (2021) 2163-2176.

[27] X. Wang, Z. Ning, L. Guo, S. Guo, X. Gao, G. Wang, Mean-field learning for edge computing in mobile blockchain networks, IEEE Trans. Mob. Comput. 22 (10) (2023) 5978-5994.

[28] Y. Zuo, S. Jin, S. Zhang, Y. Han, K.-K. Wong, Delay-limited computation offloading for mec-assisted mobile blockchain networks, IEEE Trans. Commun. 69 (12) (2021) 8569-8584.

[29] Y. Zuo, S. Jin, S. Zhang, Computation offloading in untrusted mec-aided mobile blockchain iot systems, IEEE Trans. Wirel. Commun. 20 (12) (2021) 8333-8347.

[30] L. Ye, J. Luo, C. Jiang, L. Gao,Collaborative block mining and edge task offload-ing in mec-assisted blockchain networks: a coalition game-theoretic approach,in:Proceedings of the 2023 IEEE Global Communications Conference, IEEE, 2023, pp. 4638-4643.

[31] Y. Liang, Y. Li, J. Guo, Y. Li, Resource competition in blockchain networks under cloud and device enabled participation, IEEE Access 10 (2022) 11979-11993.

[32] W. Guo, Z. Chang, X. Guo, P. Wu, Z. Han, Incentive mechanism for edge computing-based blockchain: a sequential game approach, IEEE Trans. Ind. Inform. 18 (11) (2022) 7899-7909.

[33] Y. Zuo, S. Jin, S. Zhang, Y. Zhang, Blockchain storage and computation offloading for cooperative mobile-edge computing, IEEE Int. Things J. 8 (11) (2021) 9084-9098.

[34] Y. Li, Z. Lin, W. Zhang, Y. Zheng, J. Yang, Mobile edge computing-enabled blockchain: contract-guided computation offloading, J. Supercomput. 79 (7) (2023) 7970-7996.

[35] L. Qi, J. Tian, M. Chai, H. Cai, Lightpow: a trust based time-constrained pow for blockchain in Internet of Things, Comput. Netw. 220 (2023) 109480.

[36] Y. Liu, K. Wang, Y. Lin, W. Xu, 𝖫𝗂𝗀𝗁𝗍𝖢𝗁𝖺𝗂𝗇: a lightweight blockchain system for in-dustrial Internet of things, IEEE Trans. Ind. Inform. 15 (6) (2019) 3571-3581.

[37] J. Xu, L. Xu, Optimal stochastic process optimizer: a new metaheuristic al-gorithm with adaptive exploration-exploitation property, IEEE Access 9 (2021) 108640-108664.

[38] U. Syed, M. Bowling, R.E. Schapire,Apprenticeship learning using linear program-ming, in:Proceedings of the 25th International Conference on Machine Learning, ACM, 2008, pp. 1032-1039.

[39] J.R. Sampson, Adaptation in natural and artificial systems (John H. Holland), SIAM Rev. 18 (3) (1976) 529-530.

[40] J. Xue, B. Shen, A novel swarm intelligence optimization approach: sparrow search algorithm, Syst. Sci. Control Eng. 8 (1) (2020) 22-34.

[41] D. Whitley, A genetic algorithm tutorial, Stat. Comput. 4 (1994) 65-85.

[42] A.W. Mohamed, A.A. Hadi, K.M. Jambi, Novel mutation strategy for enhancing shade and lshade algorithms for global numerical optimization, Swarm Evol. Comput. 50 (2019) 100455.

[43] Y. Shi, Particle swarm optimization, IEEE Connect. 2 1 ( 2004) 8-13.

[44] S. Mirjalili, S.M. Mirjalili, A. Lewis, Grey wolf optimizer, Adv. Eng. Softw. 69 (2014) 46-61.

[45] A.A. Heidari, S. Mirjalili, H. Faris, I. Aljarah, M. Mafarja, H. Chen, Harris hawks optimization: algorithm and applications, Future Gener. Comput. Syst. 97 (2019) 849-872.

[46] A. Faramarzi, M. Heidarinejad, S. Mirjalili, A.H. Gandomi, Marine predators algo-rithm: a nature-inspired metaheuristic, Expert Syst. Appl. 152 (2020) 113377.

[47] A. Seyyedabbasi, F. Kiani, Sand cat swarm optimization: a nature-inspired algorithm to solve global optimization problems, Eng. Comput. 39 (4) (2023) 2627-2651.

[48] L. Wang, Q. Cao, Z. Zhang, S. Mirjalili, W. Zhao, Artificial rabbits optimization: a new bio-inspired meta-heuristic algorithm for solving engineering optimization problems, Eng. Appl. Artif. Intell. 114 (2022) 105082.

[49] F.A. Hashim, E.H. Houssein, K. Hussain, M.S. Mabrouk, W. Al-Atabany, Honey badger algorithm: new metaheuristic algorithm for solving optimization problems, Math. Comput. Simul. 192 (2022) 84-110.

[50] B. Abdollahzadeh, F.S. Gharehchopogh, S. Mirjalili, African vultures optimization algorithm: a new nature-inspired metaheuristic algorithm for global optimization problems, Comput. Ind. Eng. 158 (2021) 107408.

[51] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, M. Riedmiller,Deterministic policy gradient algorithms, in:Proceedings of the 31st International Conference on Machine Learning, vol. 32, PMLR, 2014, pp. 387-395.

[52] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine,Soft actor-critic: off-policy maximum en-tropy deep reinforcement learning with a stochastic actor, in: Proceedings of the 35th International Conference on Machine Learning,in:Proceedings of Machine Learning Research, vol. 80, PMLR, 2018, pp. 1861-1870.