CHPFL: Clustered adaptive hierarchical federated learning for edge-level personalization

Abstract

Federated learning faces challenges with non-IID data distributions, often resulting in suboptimal performance for individual clients with the global model. To address this issue, we propose a clustered hierarchical personalized federated learning (CHPFL) framework, which provides edge-level personalization to effectively overcomes non-IID data and alleviates the overfitting in the personalization process. The three-layer framework makes the learning and personalization process more feasible compared to traditional two-layer federated learning, as edge servers typically offer greater computing power and more efficient communication with the cloud server. Specifically, we use the K-Means++ clustering algorithm to group local clients based on their model updates, ensuring that clients with similar data distributions are clustered together and assigned to the same edge server. Each edge server then generates a personalized model by blending the global model with the edge model, which is adaptively updated and optimized through multiple iterations. Additionally, we introduce a novel aggregation rule on the cloud server to produce a global model with improved performance. Experiments on the MNIST, FMNIST, and KMNIST datasets demonstrate that CHPFL effectively overcomes non-IID data distribution and outperforms HPFL, APFL, and FedALA in non-IID settings.

Keywords

Federated learning / Personalized federated learning / Hierarchical federated learning / Non-IID data

CRediT authorship contribution statement

Lihua Song: Funding acquisition. Jing Li: Validation, Project administration, Supervision. Honglu Jiang: Methodology. Shuhua Wei: Resources. Yufei Guo: Writing - review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

Lihua Song is supported by National Key R&D Program of China (2024YFE0200500).

References

[1] Pariwat Ongsulee, Artificial intelligence, machine learning and deep learning, in: 2017 15th International Conference on ICT and Knowledge Engineering (ICT&KE), IEEE, 2017, pp. 1-6.

[2] Sawsan Abdulrahman, Hanine Tout, Hakima Ould-Slimane, Azzam Mourad, Chamseddine Talhi, Mohsen Guizani, A survey on federated learning: The journey from centralized to distributed on-site learning and beyond, IEEE Internet Things J. 8 (7) (2021) 5476-5497.

[3] Brendan McMahan, Daniel Ramage, Federated learning: Collaborative machine learning without centralized training data, Google Res. Blog 3 (2017).

[4] Zuobin Xiong, Zhipeng Cai, Daniel Takabi, Wei Li, Privacy threat and defense for federated learning with non-i.i.d. Data in aIoT, IEEE Trans. Ind. Informatics PP (99) (2021) 1.

[5] Yan Ding, Kenli Li, Chubo Liu, Keqin Li, A potential game theoretic approach to computation offloading strategy optimization in end-edge-cloud computing, IEEE Trans. Parallel Distrib. Syst. 33 (6) (2021) 1503-1519.

[6] Caihong Kai, Hao Zhou, Yibo Yi, Wei Huang, Collaborative cloud-edgeend task offloading in mobile-edge computing networks with limited communication capability, IEEE Trans. Cogn. Commun. Netw. 7 (2) (2020) 624-634.

[7] Qi Xia, Winson Ye, Zeyi Tao, Jindi Wu, Qun Li, A survey of federated learning for edge computing: Research problems and solutions, High-Confid. Comput. (2021).

[8] Qiong Wu, Kaiwen He, Xu Chen, Personalized federated learning for intelligent IoT applications: A cloud-edge based framework, IEEE Open J. Comput. Soc. 1 (2020) 35-44.

[9] Zuobin Xiong, Wei Li, Zhipeng Cai, Federated generative model on multisource heterogeneous data in iot, in:Proceedings of the AAAI Conference on Artificial Intelligence, 37, (9) 2023, pp. 10537-10545.

[10] Junjie Pang, Yan Huang, Zhenzhen Xie, Qilong Han, Zhipeng Cai, Realizing the heterogeneity: A self-organized federated learning framework for IoT, IEEE Internet Things J. 8 (5) (2021) 3088-3098.

[11] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning, Found. Trends® Mach. Learn. 14 (1-2) (2021) 1-210.

[12] Zuobin Xiong, Zhipeng Cai, Daniel Takabi, Wei Li, Privacy threat and defense for federated learning with non-iid data in aIoT, IEEE Trans. Ind. Informatics 18 (2) (2021) 1310-1321.

[13] Daliang Li, Junpu Wang, Fedmd: Heterogenous federated learning via model distillation, 2019, arXiv preprint arXiv:1910.03581.

[14] Yuyang Deng, Mohammad Mahdi Kamani, Mehrdad Mahdavi, Adaptive personalized federated learning, 2020, arXiv preprint arXiv:2003.13461.

[15] Chuxiao Su, Jing Wu, Rui Zhang, Zi Kang, Hui Xia, Cheng Zhang, FedBS: Solving data heterogeneity issue in federated learning using bal-anced subtasks, High-Confid. Comput. (2025) 100322, URL https://www.sciencedirect.com/science/article/pii/S2667295225000261.

[16] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, Ameet S Talwalkar, Federated multi-task learning, Adv. Neural Inf. Process. Syst. 30 (2017).

[17] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith, Federated optimization in heterogeneous networks, Proc. Mach. Learn. Syst. 2 (2020) 429-450.

[18] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, Ananda Theertha Suresh, Scaffold: Stochastic controlled averaging for federated learning, in: International Conference on Machine Learning, PMLR, 2020, pp. 5132-5143.

[19] Alireza Fallah, Aryan Mokhtari, Asuman Ozdaglar, Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach, Adv. Neural Inf. Process. Syst. 33 (2020) 3557-3568.

[20] Bingyan Liu, Yao Guo, Xiangqun Chen, PFA: Privacy-preserving federated adaptation for effective model personalization, in: Proceedings of the Web Conference 2021, 2021, pp. 923-934.

[21] Filip Hanzely, Peter Richtárik, Federated learning of a mixture of global and local models, 2020, arXiv preprint arXiv:2002.05516.

[22] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, Amos Storkey, Meta-learning in neural networks: A survey, IEEE Trans. Pattern Anal. Mach. Intell. 44 (9) (2021) 5149-5169.

[23] Yihan Jiang, Jakub Konečn`y, Keith Rush, Sreeram Kannan, Improving federated learning personalization via model agnostic meta learning, 2019, arXiv preprint arXiv:1909.12488.

[24] Mikhail Khodak, Maria-Florina F. Balcan, Ameet S. Talwalkar, Adaptive gradient-based meta-learning methods, Adv. Neural Inf. Process. Syst. 32 (2019).

[25] Mahdi Morafah, Saeed Vahidian, Weijia Wang, Bill Lin, FLIS: Clustered federated learning via inference similarity for non-iid data distribution, IEEE Open J. Comput. Soc. 4 (2023) 109-120.

[26] Zaobo He, Lintao Wang, Zhipeng Cai, Clustered federated learning with adaptive local differential privacy on heterogeneous IoT data, IEEE Internet Things J. (2023) 1.

[27] Mahdi Morafah, Saeed Vahidian, Weijia Wang, Bill Lin, Flis: Clustered federated learning via inference similarity for non-iid data distribution, IEEE Open J. Comput. Soc. 4 (2023) 109-120.

[28] Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, Yujuan Tan, FedGroup: Accurate federated learning via decomposed similarity-based clustering, 2020.

[29] Avishek Ghosh, Justin Hong, Dong Yin, Kannan Ramchandran, Robust federated learning in a heterogeneous environment, 2019.

[30] Shaohua Cao, Hanqing Zhang, Tian Wen, Hongwei Zhao, Quancheng Zheng, Weishan Zhang, Danyang Zheng, FedQMIX: Communication-efficient fed-erated learning via multi-agent reinforcement learning, High- Confid. Comput. 4 (2) (2024) 100179, URL https://www.sciencedirect.com/science/article/pii/S2667295223000776.

[31] Ruiqi Liu, Songcan Yu, Linsi Lan, Junbo Wang, Krishna Kant, Neville Calleja, A remedy for heterogeneous data: Clustered federated learning with gradient trajectory, Big Data Min. Anal. 7 (4) (2024) 1050-1064, URL https://www.sciopen.com/article/10.26599/BDMA.2024.9020065.

[32] Zaobo He, Lintao Wang, Zhipeng Cai, Clustered federated learning with adaptive local differential privacy on heterogeneous iot data, IEEE Internet Things J. 11 (1) (2023) 137-146.

[33] Chuantao Li, Bruce Gu, Zhigang Zhao, Youyang Qu, Guomao Xin, Jidong Huo, Longxiang Gao, Federated transfer learning for on-device LLMs effi-cient fine tuning optimization, Big Data Min. Anal. 8 (2) (2025) 430-446, URL https://www.sciopen.com/article/10.26599/BDMA.2024.9020068.

[34] Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, Wen Gao, Fedhealth: A federated transfer learning framework for wearable healthcare, IEEE Intell. Syst. 35 (4) (2020) 83-93.

[35] Hongwei Yang, Hui He, Weizhe Zhang, Xiaochun Cao, FedSteg: A federated transfer learning framework for secure image steganalysis, IEEE Trans. Netw. Sci. Eng. 8 (2) (2020) 1084-1094.

[36] Rui Zhao, Peng Zhi, Xiao Yang, Zhihe Zhang, Gang Liu, Changyan Di, Qingguo Zhou, Client to server: Heterogeneous distribution knowledge transfer for federated learning, Tsinghua Sci. Technol. (2025) URL https://www.sciopen.com/article/10.26599/TST.2025.9010047.

[37] Ertong Shang, Hui Liu, Jingyang Zhang, Runqi Zhao, Junzhao Du, Ensemble knowledge distillation for federated semi-supervised image classification, Tsinghua Sci. Technol. 30 (1) (2025) 112-123, URL https://www.sciopen.com/article/10.26599/TST.2023.9010156.

[38] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, Yong Zhang, Personalized cross-silo federated learning on non-iid data, in:Proceedings of the AAAI Conference on Artificial Intelligence, 35, (9) 2021, pp. 7865-7873.

[39] Yishay Mansour, Mehryar Mohri, Jae Ro, Ananda Theertha Suresh, Three approaches for personalization with applications to federated learning, 2020, arXiv preprint arXiv:2002.10619.

[40] Xiaofeng Liu, Yinchuan Li, Yunfeng Shao, Qing Wang, Sparse federated learning with hierarchical personalized models, 2022, arXiv:2203.13517.

[41] Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, Haibing Guan, Fedala: Adaptive local aggregation for personalized federated learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, 37, (9) 2023, pp. 11237-11244.

[42] David Arthur, Sergei Vassilvitskii, K-Means++: The advantages of careful seeding, in: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2007, New Orleans, Louisiana, USA, January 7-9, 2007, 2007.

[43] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, Gradient-based learning applied to document recognition, Proc. IEEE 86 (11) (1998) 2278-2324.

[44] Han Xiao, Kashif Rasul, Roland Vollgraf, Fashion-mnist: a novel im-age dataset for benchmarking machine learning algorithms, 2017, arXiv preprint arXiv:1708.07747.

[45] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, David Ha, Deep learning for classical japanese literature, 2018, arXiv preprint arXiv:1812.01718.

[46] Lihua Song, Jing Li, Honglu Jiang, Shuhua Wei, Yufei Guo, Adaptive edge-level personalization on hierarchical federated learning, in: 2023 IEEE International Performance, Computing, and Communications Conference, IPCCC, IEEE, 2023, pp. 397-402.