Device-edge collaborative occluded face recognition method based on cross-domain feature fusion

Abstract

The lack of facial features caused by wearing masks degrades the performance of facial recognition systems. Traditional occluded face recognition methods cannot integrate the computational resources of the edge layer and the device layer. Besides, previous research fails to consider the facial characteristics including occluded and unoccluded parts. To solve the above problems, we put forward a device-edge collaborative occluded face recognition method based on cross-domain feature fusion. Specifically, the device-edge collaborative face recognition architecture gets the utmost out of maximizes device and edge resources for real-time occluded face recognition. Then, a cross-domain facial feature fusion method is presented which combines both the explicit domain and the implicit domain facial. Furthermore, a delay-optimized edge recognition task scheduling method is developed that comprehensively considers the task load, computational power, bandwidth, and delay tolerance constraints of the edge. This method can dynamically schedule face recognition tasks and minimize recognition delay while ensuring recognition accuracy. The experimental results show that the proposed method achieves an average gain of about 21% in recognition latency, while the accuracy of the face recognition task is basically the same compared to the baseline method.

Keywords

Occluded face recognition / Cross-domain feature fusion / Device-edge collaboration

CRediT authorship contribution statement

Puning Zhang: Conceptualization, Data curation, Formal analysis. Lei Tan: Formal analysis, Investigation. Zhigang Yang: Investigation, Methodology. Fengyi Huang: Resources, Software. Lijun Sun: Validation, Visualization. Haiying Peng: Supervision, Validation.

Declaration of Competing Interest

The authors have no conflict of interest.

Acknowledgements

This work is supported by National Natural Science Foundation of China (61901071, 61871062, 61771082, U20A20157), Science and Natural Science Foundation of Chongqing, China (cstc2020jcyj-zdxmX0024), University Innovation Research Group of Chongqing (CXQT20017), Program for Innovation Team Building at Institutions of Higher Education in Chongqing(CXTDX201601020), Natural Science Foundation of Chongqing, China (CSTB2022NSCQ-MSX0600), and Youth Innovation Group Support Program of ICE Discipline of CQUPT (SCIE-QN-2022-04). Chongqing Municipal Technology Innovation and Application Development Special Key Project (cstc2020jscx-dxwtBX0053), China Postdoctoral Science Foundation Project, China (2022MD723723), Chongqing Postdoctoral Research Project Special Funding, China (2023CQBSHTB3092).

References

[1] G. Guo, N. Zhang, A survey on deep learning based face recognition, Comput. Vis. Image Underst. 189 (2019) 102805.

[2] I. Adjabi, A. Ouahabi, A. Benzaoui, A. Taleb-Ahmed, Past, present, and future of face recognition: A review, Electronics 9 (8) (2020) 1188.

[3] N.H. Barnouti, et al., Improve face recognition rate using different image pre-processing techniques, Am. J. Eng. Res. 5 (4) (2016) 46-53.

[4] M.S. Ejaz, M.R. Islam, M. Sifatullah, A. Sarker, Implementation of principal compo-nent analysis on masked and non-masked face recognition, in: 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICAS-ERT), IEEE, 2019, pp. 1-5.

[5] C. Liu, H. Wechsler, Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition, IEEE Trans. Image Process. 11 (4) (2002) 467-476.

[6] Y. Xu, A. Zhong, J. Yang, D. Zhang, Lpp solution schemes for use with face recogni-tion, Pattern Recognit. 43 (12) (2010) 4165-4176.

[7] H. Ben Fredj, S. Bouguezzi, C. Souani, Face recognition in unconstrained environ-ment with cnn, Vis. Comput. 37 (2) (2021) 217-226.

[8] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint, arXiv :1409.1556.

[9] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, L. Song, Sphereface: Deep hypersphere em-bedding for face recognition,in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 212-220.

[10] F. Schroff, D. Kalenichenko, J. Philbin, Facenet: A unified embedding for face recog-nition and clustering, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 815-823.

[11] G. Jeevan, G.C. Zacharias, M.S. Nair, J. Rajan, An empirical study of the impact of masks on face recognition, Pattern Recognit. 122 (2022) 108308.

[12] J. Deng, J. Guo, N. Xue, S. Zafeiriou, Arcface: Additive angular margin loss for deep face recognition,in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4690-4699.

[13] F. Ding, P. Peng, Y. Huang, M. Geng, Y. Tian,Masked face recognition with latent part detection, in:Proceedings of the 28th ACM international Conference on multi-media, 2020, pp. 2281-2289.

[14] L. Feihong, C. Hang, L. Kang, D. Qiliang, Z. Jian, Z. Kaipeng, H. Hong, Toward high-quality face-mask occluded restoration, ACM Trans. Multimed. Comput. Commun. Appl. 19 (1) (2023) 1-23.

[15] W. Shi, J. Cao, Q. Zhang, Y. Li, L. Xu, Edge computing: Vision and challenges, IEEE Int. Things J. 3 (5) (2016) 637-646.

[16] S. Woo, J. Park, J.-Y. Lee, I.S. Kweon, Cbam: Convolutional block attention mod-ule,in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3-19.

[17] Y. Zhang, X. Wang, M.S. Shakeel, H. Wan, W. Kang, Learning upper patch attention using dual-branch training strategy for masked face recognition, Pattern Recognit. 126 (2022) 108522.

[18] T. Tang, L. Li, X. Wu, R. Chen, H. Li, G. Lu, L. Cheng, Tsa-scc: text semantic-aware screen content coding with ultra low bitrate, IEEE Trans. Image Process. 31 (2022) 2463-2477.

[19] H.N. Vu, M.H. Nguyen, C. Pham, Masked face recognition with convolutional neural networks and local binary patterns, Appl. Intell. 52 (5) (2022) 5497-5512.

[20] B. Yin, L. Tran, H. Li, X. Shen, X. Liu,Towards interpretable face recognition, in:Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9348-9357.

[21] G. Gao, J. Yang, X.-Y. Jing, F. Shen, W. Yang, D. Yue, Learning robust and dis-criminative low-rank representations for face recognition with occlusion, Pattern Recognit. 66 (2017) 129-143.

[22] K. Zhang, W. Zuo, S. Gu, L. Zhang, Learning deep cnn denoiser prior for image restoration, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3929-3938.

[23] Y. Zeng, J. Fu, H. Chao, B. Guo,Learning pyramid-context encoder network for high-quality image inpainting, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1486-1494.

[24] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, A.A. Efros, Context encoders: Feature learning by inpainting, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2536-2544.

[25] D. Wu, X. Han, Z. Yang, R. Wang, Exploiting transfer learning for emotion recog-nition under cloud-edge-client collaborations, IEEE J. Sel. Areas Commun. 39 (2) (2020) 479-490.

[26] A. Meslin, N. Rodriguez, M. Endler, Scalable mobile sensing for smart cities: The musanet experience, IEEE Int. Things J. 7 (6) (2020) 5202-5209.

[27] D. Wu, R. Bao, Z. Li, H. Wang, H. Zhang, R. Wang, Edge-cloud collaboration en-abled video service enhancement: a hybrid human-artificial intelligence scheme, IEEE Trans. Multimed. 23 (2021) 2208-2221.

[28] X. Zhou, X. Xu, W. Liang, Z. Zeng, Z. Yan, Deep-learning-enhanced multitarget detec-tion for end-edge-cloud surveillance in smart iot, IEEE Int. Things J. 8 (16) (2021) 12588-12596.

[29] H. Sun, W. Shi, X. Liang, Y. Yu, Vu: edge computing-enabled video usefulness detec-tion and its application in large-scale video surveillance systems, IEEE Int. Things J. 7 (2) (2019) 800-817.

[30] L. Song, D. Gong, Z. Li, C. Liu, W. Liu,Occlusion robust face recognition based on mask learning with pairwise differential siamese network, in:Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 773-782.

[31] Y. Li,Deep reinforcement learning: An overview, arXiv preprint, arXiv :1701.07274.

[32] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy opti-mization algorithms, arXiv preprint, arXiv :1707.06347.

[33] G.B. Huang, M. Mattar, T. Berg, E. Learned-Miller, Labeled faces in the wild: A database for studying face recognition in unconstrained environments,in:Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, 2008.

[34] A. Anwar, A. Raychowdhury, Masked face recognition for secure authentication, arXiv preprint, arXiv : 2008.11104.

[35] S. Targ, D. Almeida, K. Lyman, Resnet in resnet: Generalizing residual architectures, arXiv preprint, arXiv :1603.08029.

[36] M.A. Wani, F.A. Bhat, S. Afzal, A.I. Khan, Supervised deep learning in face recogni-tion, in: Advances in Deep Learning, Springer, 2020, pp. 95-110.

[37] P.C. Neto, F. Boutros, J.R. Pinto, N. Darner, A.F. Sequeira, J.S. Cardoso, Focusface: Multi-task contrastive learning for masked face recognition, in: 2021 16th IEEE In-ternational Conference on Automatic Face and Gesture Recognition (FG 2021), IEEE, 2021, pp. 01-08.