A privacy-preserving class imbalance mitigation framework for face recognition

Abstract

AI-powered face recognition has become essential to various IoT applications, including home automation, security systems, and personalized services. While these systems offer significant advancements, they still face critical challenges related to accuracy and privacy. One major issue is class imbalance, which is common in face recognition systems where certain demographic groups are underrepresented. This imbalance results in biased models, compromising the accuracy and fairness of these systems. Furthermore, traditional centralized training methods can expose sensitive facial data, raising serious privacy concerns. Federated Learning (FL) has emerged as a solution to improve model training by enabling collaboration across devices without sharing sensitive data. However, it also worsens the issue of data heterogeneity. This paper proposes a Hierarchical Federated Learning (HFL) framework to address class imbalance while preserving privacy. By aggregating local models at different hierarchical levels, the framework mitigates data imbalance and enhances fairness in face recognition systems. Additionally, a privacy-preserving mechanism based on Secure Multi-Party Computation (SMPC) is implemented to ensure data security during the training process.

Keywords

Class imbalance / Hierarchical federated learning / Privacy / Secure Multi-Party Computation

CRediT authorship contribution statement

Amani Aldahiri: Writing - original draft, Conceptualization. Ibrahim Khalil: Supervision. Mohammad Saidur Rahman: Supervision. Mohammed Atiquzzaman: Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This work was supported by the Australian Research Council Discovery Project (DP220100215).

References

[1] S. Ghafoor, K.B. Khan, M.R. Tahir, M. Mustafa, Home automation security system based on face detection and recognition using IoT, in: Intelligent Technologies and Applications: Second International Conference, INTAP 2019, Bahawalpur, Pakistan, November 6-8, 2019, Revised Selected Papers 2, Springer, 2020, pp. 67-78.

[2] I.M. Sayem, M.S. Chowdhury, Integrating face recognition security system with the internet of things, in: 2018 International Conference on Machine Learning and Data Engineering, ICMLDE, IEEE, 2018, pp. 14-18.

[3] I. Walter, J. Ney, T. Hotfilter, V. Rybalkin, J. Hoefer, N. Wehn, J. Becker, Embedded face recognition for personalized services in the assistive robotics, in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, 2021, pp. 339-350.

[4] Q. Meng, S. Zhao, Z. Huang, F. Zhou, Magface: A universal representation for face recognition and quality assessment, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14225-14234.

[5] M. Gentzel, Biased face recognition technology used by government: A problem for liberal democracy, Philos. Technol. 34 (4) (2021) 1639-1663.

[6] A. Guzmán-Ponce, J.S. Sánchez, R.M. Valdovinos, J.R. Marcial-Romero, DBIG-US: A two-stage under-sampling algorithm to face the class imbalance problem, Expert Syst. Appl. 168 (2021) 114301.

[7] S. Lu, Z. Gao, Q. Xu, C. Jiang, A. Zhang, X. Wang, Class-imbalance privacy-preserving federated learning for decentralized fault diagnosis with biometric authentication, IEEE Trans. Ind. Inform. 18 (12) (2022) 9101-9111.

[8] V. Pandimurugan, A. Jain, Y. Sinha, IoT based face recognition for smart applications using machine learning, in: 2020 3rd International Conference on Intelligent Sustainable Systems, ICISS, IEEE, 2020, pp. 1263-1266.

[9] S. Gong, X. Liu, A.K. Jain, Jointly de-biasing face recognition and demographic attribute estimation, in: Computer Vision-ECCV 2020: 16 th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, Springer, 2020, pp. 330-347.

[10] Z. Shen, J. Cervino, H. Hassani, A. Ribeiro, An agnostic approach to federated learning with class imbalance, in:International Conference on Learning Representations, 2021.

[11] J.M. Johnson, T.M. Khoshgoftaar, Survey on deep learning with class imbalance, J. Big Data 6 (1) (2019) 1-54.

[12] L. Wang, S. Xu, X. Wang, Q. Zhu, Addressing class imbalance in federated learning, in:Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 10165-10173.

[13] L. Liu, J. Zhang, S. Song, K.B. Letaief, Hierarchical federated learning with quantization: Convergence analysis and system design, IEEE Trans. Wirel. Commun. 22 (1) (2022) 2-18.

[14] Y. Cui, K. Cao, J. Zhou, T. Wei, Optimizing training efficiency and cost of hierarchical federated learning in heterogeneous mobile-edge cloud computing, IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 42 (5) (2022) 1518-1531.

[15] A.A. Abdellatif, N. Mhaisen, A. Mohamed, A. Erbad, M. Guizani, Z. Dawy, W. Nasreddine, Communication-efficient hierarchical federated learning for IoT heterogeneous systems with imbalanced data, Future Gener. Comput. Syst. 128 (2022) 406-419.

[16] N. Bouacida, P. Mohapatra, Vulnerabilities in federated learning, IEEE Access 9 (2021) 63229-63249.

[17] C. Zhao, S. Zhao, M. Zhao, Z. Chen, C.-Z. Gao, H. Li, Y.-a. Tan, Secure multi-party computation: theory, practice and applications, Inform. Sci. 476 (2019) 357-372.

[18] W. Du, M.J. Atallah, Secure multi-party computation problems and their applications: a review and open problems, in: Proceedings of the 2001 Workshop on New Security Paradigms, 2001, pp. 13-22.

[19] Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, B. He, A survey on federated learning systems: Vision, hype and reality for data privacy and protection, IEEE Trans. Knowl. Data Eng. (2021).

[20] X. Shuai, Y. Shen, S. Jiang, Z. Zhao, Z. Yan, G. Xing, BalanceFL: Addressing class imbalance in long-tail federated learning, in: 2022 21st ACM/IEEE International Conference on Information Processing in Sensor Networks, IPSN, IEEE, 2022, pp. 271-284.

[21] Y. Dai, Z. Chen, J. Li, S. Heinecke, L. Sun, R. Xu, Tackling data heterogeneity in federated learning with class prototypes, in:Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 6, 2023, pp. 7314-7322.

[22] M. Buda, A. Maki, M.A. Mazurowski, A systematic study of the class imbalance problem in convolutional neural networks, Neural Netw. 106 (2018) 249-259.

[23] Y. Hong, S. Han, K. Choi, S. Seo, B. Kim, B. Chang, Disentangling label distribution for long-tailed visual recognition, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6626-6636.

[24] J. Ren, C. Yu, X. Ma, H. Zhao, S. Yi, et al., Balanced meta-softmax for long-tailed visual recognition, Adv. Neural Inf. Process. Syst. 33 (2020) 4175-4186.

[25] B. Zhou, Q. Cui, X.-S. Wei, Z.-M. Chen, Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9719-9728.

[26] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, S.X. Yu, Large-scale long-tailed recognition in an open world, in:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2537-2546.

[27] S.P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, A.T. Suresh, Scaffold: Stochastic controlled averaging for federated learning, in: International Conference on Machine Learning, PMLR, 2020, pp. 5132-5143.

[28] F. Sattler, S. Wiedemann, K.-R. Müller, W. Samek, Robust and communication-efficient federated learning from non-iid data, IEEE Trans. Neural Netw. Learn. Syst. 31 (9) (2019) 3400-3413.

[29] M. Duan, D. Liu, X. Chen, R. Liu, Y. Tan, L. Liang, Self-balancing federated learning with global imbalanced data in mobile systems, IEEE Trans. Parallel Distrib. Syst. 32 (1) (2020) 59-71.

[30] O. Marfoq, G. Neglia, R. Vidal, L. Kameni, Personalized federated learning through local memorization, in: Proc. Int. Conf. Mach. Learn., 2022, pp. 15070-15092.

[31] T. Li, A.K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, V. Smith, Federated optimization in heterogeneous networks, Proc. Mach. Learn. Syst. 2 (2020) 429-450.

[32] A. Ghosh, J. Chung, D. Yin, K. Ramchandran, An efficient framework for clustered federated learning, in: Proc. Adv. Neural Inf. Process. Syst., 2020, pp. 1-20.

[33] F. Sattler, K.-R. Müller, W. Samek, Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints, IEEE Trans. Neural Netw. Learn. Syst. 32 (8) (2021) 3710-3722.

[34] H. Huang, W. Shi, Y. Feng, C. Niu, Active client selection for clustered federated learning, IEEE Trans. Neural Netw. Learn. Syst. (2023) [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10197242/.

[35] J. Xu, H. Fan, Q. Wang, Y. Jiang, Q. Duan, Adaptive idle model fusion in Hierarchical Federated Learning for unbalanced edge regions, IEEE Trans. Netw. Sci. Eng. (2024).

[36] S. Luo, X. Chen, Q. Wu, Z. Zhou, S. Yu, HFEL: Joint edge association and resource allocation for cost-efficient hierarchical federated edge learning, IEEE Trans. Wirel. Commun. 19 (10) (2020) 6535-6548.

[37] C. Liu, T.J. Chua, J. Zhao, Time minimization in hierarchical federated learning, in: 2022 IEEE Global Communications Conference, GLOBECOM, IEEE, 2022, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9996746/.

[38] Z. Wang, H. Xu, J. Liu, H. Huang, C. Qiao, Resource-efficient federated learning with hierarchical aggregation in edge computing, IEEE Trans. Netw. Serv. Manag. 18 (4) (2021) 4562-4575, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9488756/.

[39] G.B. Huang, M. Mattar, T. Berg, E. Learned-Miller, Labeled faces in the wild: A database forstudying face recognition in unconstrained environments,in:Workshop on Faces in’Real-Life’Images: Detection, Alignment, and Recognition, 2008.

[40] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in:Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2980-2988.

[41] B. McMahan, E. Moore, D. Ramage, S. Hampson, B.A. y Arcas, Communication-efficient learning of deep networks from decentralized data, in: Artificial Intelligence and Statistics, PMLR, 2017, pp. 1273-1282.

[42] Y. LeCun, C. Cortes, C. Burges, et al., MNIST Handwritten Digit Database, Florham Park, NJ, USA, 2010.

[43] H. Xiao, K. Rasul, R. Vollgraf, Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017, arXiv preprint arXiv: 1708.07747.

[44] B. Knott, S. Venkataraman, A. Hannun, S. Sengupta, M. Ibrahim, L. van der Maaten, Crypten: Secure multi-party computation meets machine learning, Adv. Neural Inf. Process. Syst. 34 (2021) 4961-4973.