FedReg ∗:Addressing Non-Independent and Identically Distributed Challenges in Federated Learning

Abstract

In non-independent and identically distributed(non-IID) data environments, model performance often degrades significantly. To address this issue, two improvement methods are proposed:FedReg and FedReg^∗. FedReg is a method based on hybrid regularization aimed at enhancing federated learning in non-IID scenarios. It introduces hybrid regularization to replace traditional L2 regularization, combining the advantages of L1 and L2 regularization to enable feature selection while preventing overfitting. This method better adapts to the diverse data distributions of different clients, improving the overall model performance. FedReg^∗ combines hybrid regularization with weighted model aggregation. In addition to the benefits of hybrid regularization, FedReg^∗ applies a weighted averaging method in the model aggregation process, calculating weights based on the cosine similarity between each client gradient and the global gradient to more reasonably distribute client contributions. By considering variations in data quality and quantity among clients, FedReg^∗ highlights the importance of key clients and enhances the model’s generalization performance. These improvement methods enhance model accuracy and communication efficiency.

Keywords

federated learning / non-independent and identically distributed(non-IID) data / hybrid regularization / cosine similarity

References

[1] LI Q B, WEN Z Y, WU Z M, et al. A survey on federated learning systems:vision, hype and reality for data privacy and protection[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(4):3347-3366.

[2] WANG Y T, SU Z, PAN Y H, et al. Social-aware clustered federated learning with customized privacy preservation[J]. IEEE/ACM Transactions on Networking, 2024, 32(5):3654-3668.

[3] ZHANG T, SONG A X, DONG X W, et al. Privacy-preserving asynchronous grouped federated learning for IoT[J]. IEEE Internet of Things Journal, 2022, 9(7):5511-5523.

[4] LI L, FAN Y X, TSE M, et al. A review of applications in federated learning[J]. Computers & Industrial Engineering, 2020, 149:106854.

[5] ALEDHARI M, RAZZAK R, PARIZI R M, et al. Federated learning:a survey on enabling technologies, protocols, and applications[J]. IEEE Access, 2020, 8:140699-140725.

[6] QU Z, LI X Y, HAN X, et al. How to prevent the poor performance clients for personalized federated learning?[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). New York: IEEE, 2023:12167-12176.

[7] ZHANG H, LI C L, DAI W R, et al. Stabilizing and accelerating federated learning on heterogeneous data with partial client participation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(1):67-83.

[8] ZHONG J K, CHEN H Y, CHAO W L. Making batch normalization great in federated deep learning[EB/OL].(2023-03-12)[2024-12-30]. https://arxiv.org/abs/2303.06530v4.

[9] KAIROUZ P, MCMAHAN H B, AVENT B, et al. Advances and open problems in federated learning[EB/OL].(2021-03-09)[2024-05-30]. https://arxiv.org/abs/1912.04977.

[10] LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[J]. Proceedings of Machine Learning and Systems, 2020, 2:429-450.

[11] YANG Q, LIU Y, CHEN T J, et al. Federated machine learning[J]. ACM Transactions on Intelligent Systems and Technology, 2019, 10(2):1-19.

[12] KULKARNI P, KANHERE A, YI P H, et al. From isolation to collaboration: federated class-heterogeneous learning for chest X-ray classification[EB/OL].(2023-01-17)[2023-04-05]. https://arxiv.org/abs/2301.06683v6.

[13] WEI B J, LI J, LIU Y, et al. Federated learning for non-IID data:from theory to algorithm[C]// PRICAI 2021:Trends in Artificial Intelligence. Cham: Springer International Publishing, 2021:33-48.

[14] ZHANG F, ESTEVE-YAGÜE C, DITTMER S, et al. FedMAP: unlocking potential in personalized federated learning through bi-level MAP optimization[EB/OL].(2024-05-29)[2024-12-30]. https://arxiv.org/abs/2405.19000v1.

[15] KIM Y, SHIN B. Learning from drift: federated learning on non-IID data via drift regularization[EB/OL].(2023-09-13)[2024-12-30]. https://arxiv.org/abs/2309.07189v1.

[16] YAN Y L, FENG C M, YE M, et al. Rethinking client drift in federated learning: a logit perspective[EB/OL].(2023-08-20)[2024-12-30]. https://arxiv.org/abs/2308.10162v1.

[17] KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD:stochastic controlled averaging for federated learning[C]// International Conference on Machine Learning.[S.l.]:PMLR, 2020:5132-5143.

[18] TAN A Z, YU H, CUI L Z, et al. Towards personalized federated learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(12):9587-9603.

[19] LU Z L, PAN H, DAI Y Y, et al. Federated learning with non-IID data:a survey[J]. IEEE Internet of Things Journal, 2024, 11(11):19188-19209.

[20] WANG H, KAPLAN Z, NIU D, et al. Optimizing federated learning on non-IID data with reinforcement learning[C]// IEEE INFOCOM 2020 — IEEE Conference on Computer Communications. New York: IEEE, 2020:1698-1707.

[21] JEONG E, OH S, KIM H, et al. Communication-efficient on-device machine learning: federated distillation and augmentation under non-IID private data[EB/OL].(2018-11-28)[2024-12-30]. https://arxiv.org/abs/1811.11479v2.

[22] ZHAO Y, LI M, LAI L Z, et al. Federated learning with non-IID data[EB/OL].(2018-06-02)[2024-05-30]. https://arxiv.org/abs/1806.00582v2.

[23] PANG J J, HUANG Y, XIE Z Z, et al. Realizing the heterogeneity:a self-organized federated learning framework for IoT[J]. IEEE Internet of Things Journal, 2021, 8(5):3088-3098.

[24] JIN H, BAI D S, YAO D Z, et al. Personalized edge intelligence via federated self-knowledge distillation[J]. IEEE Transactions on Parallel and Distributed Systems, 2023, 34(2):567-580.

[25] LUO M, CHEN F, HU D P, et al. No fear of heterogeneity:lassifier calibration for federated learning with non-IID data[EB/OL].(2021-10-28)[2024-05-30]. https://arxiv.org/abs/2106.05001.

[26] SUN Y, SHEN L, CHEN S X, et al. Dynamic regularized sharpness aware minimization in federated learning:approaching global consistency and smooth landscape[C]// Proceedings of the 40th International Conference on Machine Learning.[S.l.]:PMLR, 2023:32991-33013.

[27] ACAR D A E, ZHAO Y, NAVARRO R M, et al. Federated learning based on dynamic regularization[EB/OL].(2021-11-08)[2023-04-05]. https://arxiv.org/abs/2111.04263v2.

[28] GAO L, FU H Z, LI L, et al. FedDC:federated learning with non-IID data via local drift decoupling and correction[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). New York: IEEE, 2022:10102-10111.

[29] GUO J L, LIU Z Y, LAM K Y, et al. Privacy-enhanced federated learning with weighted aggregation[C]// Security and Privacy in Social Networks and Big Data. Singapore: Springer Singapore, 2021:93-109.

[30] JI S X, TAN Y, SARAVIRTA T, et al. Emerging trends in federated learning:from model fusion to federated X learning[J]. International Journal of Machine Learning and Cybernetics, 2024, 15(9):3769-3790.

[31] XIE Z P, LIU Y, QU Z H, et al. FedALP:an adaptive layer-based approach for improved personalized federated learning[C]// Wireless Algorithms, Systems, and Applications. 17th International Conference, WASA 2022. Cham:Springer Nature, 2022, 13472:80-92.

[32] HUANG W K, YE M, SHI Z K, et al. Federated learning for generalization, robustness, fairness:a survey and benchmark[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(12):9387-9406.

[33] MA X S, ZHANG J, GUO S, et al. Layer-wised model aggregation for personalized federated learning[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). New York: IEEE, 2022:10082-10091.

[34] ZHANG F D, KUANG K, CHEN L, et al. Federated unsupervised representation learning[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(8):1181-1193.