NeurstrucEnergy: A bi-directional GNN model for energy prediction of neural networks in IoT

Abstract

A significant demand rises for energy-efficient deep neural networks to support power-limited embedding devices with successful deep learning applications in IoT and edge computing fields. An accurate energy prediction approach is critical to provide measurement and lead optimization direction. However, the current energy prediction approaches lack accuracy and generalization ability due to the lack of research on the neural network structure and the excessive reliance on customized training dataset. This paper presents a novel energy prediction model, NeurstrucEnergy. NeurstrucEnergy treats neural networks as directed graphs and applies a bi-directional graph neural network training on a randomly generated dataset to extract structural features for energy prediction. NeurstrucEnergy has advantages over linear approaches because the bi-directional graph neural network collects structural features from each layer's parents and children. Experimental results show that NeurstrucEnergy establishes state-of-the-art results with mean absolute percentage error of 2.60%. We also evaluate NeurstrucEnergy in a randomly generated dataset, achieving the mean absolute percentage error of 4.83% over 10 typical convolutional neural networks in recent years and 7 efficient convolutional neural networks created by neural architecture search. Our code is available at https://github.com/NEUSoftGreenAI/NeurstrucEnergy.git.

Keywords

Internet of things / Neural network energy prediction / Graph neural networks / Graph structure embedding / Multi-head attention

References

[1] I. Lee, K. Lee, The Internet of Things (IoT): applications, investments, and challenges for enterprises, Bus. Horiz. 58 (4) (2015) 431-440.

[2] Y. Li, C. Liao, Y. Wang, C. Wang, Energy-efficient optimal relay selection in cooperative cellular networks based on double auction, IEEE Trans. Wireless Commun. 14 (8) (2015) 4093-4104.

[3] S. Xia, Z. Yao, Y. Li, S. Mao, Online distributed offloading and computing resource management with energy harvesting for heterogeneous MEC-enabled IoT, IEEE Trans. Wireless Commun. 20 (10) (2021) 6743-6757.

[4] Y. Li, H. Ma, L. Wang, S. Mao, G. Wang, Optimized content caching and user association for edge computing in densely deployed heterogeneous networks, IEEE Trans. Mobile Comput. 21 (6) (2022) 2130-2142.

[5] X. Yao, F. Farha, R. Li, I. Psychoula, L. Chen, H. Ning, Security and privacy issues of physical objects in the IoT: challenges and opportunities, Dig. Commun. Networks 7 (3) (2021) 373-384.

[6] A. Voulodimos, N. Doulamis, A. Doulamis, E. Protopapadakis, Deep learning for computer vision: a brief review, Comput. Intell. Neurosci. 2018 (2018) 1-13.

[7] S. Hourri, N.S. Nikolov, J. Kharroubi, Convolutional neural network vectors for speaker recognition, Int. J. Speech Technol. 24 (2) (2021) 389-400.

[8] Y. Wang, J. Wang, W. Zhang, Y. Zhan, S. Guo, Q. Zheng, X. Wang, A survey on deploying mobile deep learning applications: a systemic and technical perspective, Dig. Commun. Networks 8 (1) (2022) 1-17.

[9] M. Verhelst, B. Moons, Embedded deep neural network processing: algorithmic and processor techniques bring deep learning to IoT and edge devices, IEEE Solid-State Circuits Magazine 9 (4) (2017) 55-65.

[10] S.R. Young, P. Devineni, M. Parsa, J.T. Johnston, B. Kay, R.M. Patton, C. D. Schuman, D.C. Rose, T.E. Potok, Evolving energy efficient convolutional neural networks, in: Proceedings of the IEEE International Conference on Big Data, IEEE, 2019, pp. 4479-4485.

[11] D.W. Blalock, J.J.G. Ortiz, J. Frankle, J.V. Guttag,What is the state of neural network pruning?, in:Proceedings of Machine Learning and Systems MLSYS, 2020, pp. 129-146.

[12] N. Tonellotto, A. Gotta, F.M. Nardini, D. Gadler, F. Silvestri, Neural network quantization in federated learning at the edge, Inf. Sci. 575 (2021) 417-436.

[13] X. Wang, M. Che, Y. Wei, Tensor neural network models for tensor singular value decompositions, Comput. Optim. Appl. 75 (3) (2020) 753-777.

[14] J. Gou, B. Yu, S.J. Maybank, D. Tao, Knowledge distillation: a survey, Int. J. Comput. Vis. 129 (6) (2021) 1789-1819.

[15] H. Benmeziane, K. El Maghraoui, H. Ouarnoughi, S. Niar, M. Wistuba, N. Wang, Hardware-Aware neural architecture search: survey and taxonomy,in:Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI, 2021, pp. 4322-4329.

[16] E. Cai, D.-C. Juan, D. Stamoulis, D. Marculescu, NeuralPower : predict and deploy energy-efficient convolutional neural networks,in:Proceedings of the Ninth Asian Conference on Machine Learning, PMLR, 2017, pp. 622-637.

[17] C.F. Rodrigues, G. Riley, M. Lujan, SyNERGY: an energy measurement and prediction framework for convolutional neural networks on Jetson TX,in:Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications, CSREA Press, 2018, pp. 375-382.

[18] X. Dai, Y. Jia, P. Vajda, M. Uyttendaele, N.K. Jha, P. Zhang, B. Wu, H. Yin, F. Sun, Y. Wang, M. Dukhan, Y. Hu, Y. Wu, ChamNet: towards efficient network design through platform-aware model adaptation,in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2019, pp. 11390-11399.

[19] D. Justus, J. Brennan, S. Bonner, A.S. McGough, Predicting the computational cost of deep learning models, in: Proceedings of the IEEE International Conference on Big Data, IEEE, 2018, pp. 3873-3882.

[20] D. Velasco-Montero, J. Fernandez-Berni, R. Carmona-Galan, A. Rodríguez-Vazquez, PreVIous: a methodology for prediction of visual inference performance on IoT devices, IEEE Internet Things J. 7 (10) (2020) 9227-9240.

[21] B.D. Rouhani, A. Mirhoseini, F. Koushanfar, DeLight: adding energy dimension to deep neural networks,in:Proceedings of the 2016 International Symposium on Low Power Electronics and Design, ACM, 2016, pp. 112-117.

[22] H. Qi, E.R. Sparks, A. Talwalkar, Paleo: a performance model for deep neural networks,in:Proceedings of the 5th International Conference on Learning Representations, OpenReview.net, 2017, pp. 1-10.

[23] T.-J. Yang, Y.-H. Chen, J. Emer, V. Sze, A method to estimate the energy consumption of deep neural networks, in: Proceedings of the 51st Asilomar Conference on Signals, Systems, and Computers, IEEE, 2017, pp. 1916-1920.

[24] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: Proceedings of the International Conference on Learning Representations, OpenReview.net, 2015, pp. 1-14.

[25] A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification with deep convolutional neural networks, Commun. ACM 60 (6) (2017) 84-90.

[26] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2015, pp. 1-9.

[27] G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, Densely connected convolutional networks, in: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2017, pp. 2261-2269.

[28] A.E. Eshratifar, M. Pedram, Energy and performance efficient computation offloading for deep neural networks in a mobile cloud computing environment, in: Proceedings of the 2018 on Great Lakes Symposium on VLSI, ACM, 2018, pp. 111-116.

[29] T. Elsken, J.H. Metzen, F. Hutter, Neural architecture search: a survey, J. Mach. Learn. Res. 20 (2019) 1-21.

[30] D.D. Johnson, Learning graphical state transitions, in: Proceedings of the International Conference on Learning Representations, OpenReview.net, 2017, pp. 1-19.

[31] R. Ying, R. He, K. Chen, P. Eksombatchai, W.L. Hamilton, J. Leskovec, Graph convolutional neural networks for web-scale recommender systems, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, 2018, pp. 974-983.

[32] G. Du, L. Zhou, Y. Yang, K. Lü, L. Wang, Deep multiple auto-encoder-based multi-view clustering, Data Sci. Eng. 6 (3) (2021) 323-338.

[33] Y. Peng, B. Choi, J. Xu, Graph learning for combinatorial optimization: a survey of state-of-the-art, Data Sci. Eng. 6 (2) (2021) 119-141.

[34] J. Qiu, J. Tang, H. Ma, Y. Dong, K. Wang, J. Tang, DeepInf: social influence prediction with deep learning,in:Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, 2018, pp. 2110-2119.

[35] S. Pan, R. Hu, G. Long, J. Jiang, L. Yao, C. Zhang, Adversarially regularized graph autoencoder for graph embedding, in: Proceedings of the 27th International Joint Conference on Artificial Intelligence, AAAI Press, 2018, pp. 2609-2615.

[36] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, P.S. Yu, A comprehensive survey on graph neural networks, IEEE Transact. Neural Networks Learn. Syst. 32 (1) (2021) 4-24.

[37] C. Gong, Z. Jiang, D. Wang, Y. Lin, Q. Liu, D.Z. Pan, Mixed precision neural architecture search for energy efficient deep learning, in: Proceedings of the IEEE/ACM International Conference on Computer-Aided Design, IEEE, 2019, pp. 1-7.

[38] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, ImageNet: a large-scale hierarchical image database,in:Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2009, pp. 248-255.

[39] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2016, pp. 770-778.

[40] S. Khan, M. Naseer, M. Hayat, S.W. Zamir, F.S. Khan, M. Shah, Transformers in Vision: A Survey, 2021, pp. 1-38, arXiv preprint arXiv:2101.01169.

[41] J. Mao, H. Yang, A. Li, H. Li, Y. Chen, TPrune: efficient transformer pruning for mobile devices, ACM Transact. Cyber-Phys. Syst. 5 (3) (2021) 1-22.

[42] X. Su, S. You, J. Xie, M. Zheng, F. Wang, C. Qian, C. Zhang, X. Wang, C. Xu, ViTAS: Vision Transformer Architecture Search, 2021, pp. 1-20, arXiv preprint arXiv: 2106.13700.

[43] J. Xu, X. Sun, Z. Zhang, G. Zhao, J. Lin, Understanding and improving layer normalization, in: Proceedings of the Annual Conference on Neural Information Processing Systems, Curran Associates, Inc., Vancouver, Canada, 2019, pp. 4383-4393.

[44] K. Xu, W. Hu, J. Leskovec, S. Jegelka, How powerful are graph neural networks?, in: Proceedings of the 7th International Conference on Learning Representations OpenReview. net, 2019, pp. 1-17.

[45] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, in: Proceedings of the 3rd International Conference on Learning Representations, OpenReview.net, 2015, pp. 1-15.

[46] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, Y. Bengio, Graph attention networks, in: Proceedings of the 6th International Conference on Learning Representations, OpenReview.net, 2018, pp. 1-12.

[47] D.P. Kingma, J. Ba, Adam, A method for stochastic optimization, in: Proceedings of the International Conference on Learning Representations, OpenReview.net, 2015, pp. 1-15.

[48] A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, MobileNets, Efficient Convolutional Neural Networks for Mobile Vision Applications, 2017, pp. 1-9, arXiv preprint arXiv:1704.04861.

[49] M. Sandler, A.G. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation, 2018, pp. 1-14, arXiv preprint arXiv:1801.04381.

[50] J. Redmon, A. Farhadi,YOLOv3: an Incremental Improvement, 2018, pp. 1-6, arXiv preprint arXiv:1804.02767.

[51] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, Q.V. Le, MnasNet: platform-aware neural architecture search for mobile,in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2019, pp. 2820-2828.

[52] X. Chu, B. Zhang, Q. Li, R. Xu, X. Li, Scarlet-nas: bridging the gap between stability and scalability in weight-sharing neural architecture search,in: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2021, pp. 317-325.

[53] X. Chu, B. Zhang, R. Xu, MoGA: searching beyond mobilenetv3,in:Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2020, pp. 4042-4046.

[54] S. Xie, R.B. Girshick, P. Dollar, Z. Tu, K. He, Aggregated residual transformations for deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2017, pp. 5987-5995.

[55] A. Howard, R. Pang, H. Adam, Q.V. Le, M. Sandler, B. Chen, W. Wang, L.-C. Chen, M. Tan, G. Chu, V. Vasudevan, Y. Zhu, Searching for MobileNetV3, in: Proceedings of the IEEE International Conference on Computer Vision, IEEE, 2019, pp. 1314-1324.

[56] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L.u. Kaiser, I. Polosukhin, Attention is all you need, in:Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017, pp. 5998-6008.

[57] Z. Lu, S. Rallapalli, K. Chan, S. Pu, T.L. Porta, Augur: modeling the resource requirements of ConvNets on mobile devices, IEEE Trans. Mobile Comput. 20 (2)(2021) 352-365.

[58] W.L. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs, in: Proceedings of the Annual Conference on Neural Information Processing Systems, Curran Associates, Inc., 2017, pp. 1024-1034.