FedQMIX: Communication-efficient federated learning via multi-agent reinforcement learning

Abstract

Since the data samples on client devices are usually non-independent and non-identically distributed (non-IID), this will challenge the convergence of federated learning (FL) and reduce communication efficiency. This paper proposes FedQMIX, a node selection algorithm based on multi-agent reinforcement learning(MARL), to address these challenges. Firstly, we observe a connection between model weights and data distribution, and a clustering algorithm can group clients with similar data distribution into the same cluster. Secondly, we propose a QMIX-based mechanism that learns to select devices from clustering results in each communication round to maximize the reward, penalizing the use of more communication rounds and thereby improving the communication efficiency of FL. Finally, experiments show that FedQMIX can reduce the number of communication rounds by 11% and 30% on the MNIST and CIFAR-10 datasets, respectively, compared to the baseline algorithm (Favor).

Keywords

Communication efficient / Federated learning / MARL

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This work was supported by the National Natural Science Foundation of China (NSFC) (62072469).

References

[1] Y. Gong, X. Li, L. Wang, FedMBC: Personalized federated learning via mutually beneficial collaboration, Comput. Commun. 205 (2023) 108-117.

[2] N. Abbas, Z. Yan, A. Taherkordi, T. Skeie, Mobile edge computing: A survey, IEEE Internet Things J. PP (99) (2017) 1.

[3] Q. Li, Y. Diao, Q. Chen, B. He, Federated learning on non-IID data silos: An experimental study, in: 2022 IEEE 38th International Conference on Data Engineering, 2022, pp. 965-978.

[4] B. McMahan, E. Moore, D. Ramage, S. Hampson, B.A.y. Arcas, Communication-efficient learning of deep networks from decentralized data, in: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics,in:Proceedings of Machine Learning Research, vol. 54, PMLR, 2017, pp. 1273-1282.

[5] J. Hamer, M. Mohri, A.T. Suresh, Fedboost: A communication-efficient algorithm for federated learning,in:International Conference on Machine Learning, PMLR, 2020, pp. 3973-3983.

[6] Y. Wang, Z. Su, N. Zhang, A. Benslimane, Learning in the air: Secure federated learning for UAV-assisted crowdsensing, IEEE Trans. Netw. Sci. Eng. 8 (2) (2021) 1055-1069.

[7] H. Yang, H. He, W. Zhang, X. Cao, FedSteg: A federated transfer learning framework for secure image steganalysis, IEEE Trans. Netw. Sci. Eng. 8 (2) (2021) 1084-1094.

[8] K. Cheng, T. Fan, Y. Jin, Y. Liu, T. Chen, D. Papadopoulos, Q. Yang, SecureBoost: A lossless federated learning framework, IEEE Intell. Syst. 36 (6) (2021) 87-98.

[9] S. Yang, H. Park, J. Byun, C. Kim, Robust federated learning with noisy labels, IEEE Intell. Syst. 37 (2) (2022) 35-43.

[10] F. Sattler, S. Wiedemann, K. M., W. Samek, Robust and communication-efficient federated learning from non-iid data, IEEE Trans. Neural Netw. Learn. Syst. 31 (9) (2019) 3400-3413.

[11] H. Liu, F. He, G. Cao, Communication-efficient federated learning for heterogeneous edge devices based on adaptive gradient quantization, in: IEEE INFOCOM 2023-IEEE Conference on Computer Communications, IEEE, 2023, pp. 1-10.

[12] C. Wu, F. Wu, L. Lyu, Y. Huang, X. Xie, Communication-efficient federated learning via knowledge distillation, Nat. Commun. 13 (1) (2022) 2032.

[13] X. Li, K. Huang, W. Yang, S. Wang, Z. Zhang, On the convergence of FedAvg on non-IID data, in: 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020.

[14] J. Zhang, S. Guo, Z. Qu, D. Zeng, Y. Zhan, Q. Liu, R. Akerkar, Adaptive federated learning on non-iid data with resource constraint, IEEE Trans. Comput. 71 (7) (2021) 1655-1667.

[15] K. Wang, X. Ye, K. Sakurai, Federated learning with clustering-based partic-ipant selection for IoT applications, in: 2022 IEEE International Conference on Big Data, 2022, pp. 6830-6831.

[16] H. Jamali-Rad, M. Abdizadeh, A. Singh, Federated learning with taskonomy for non-IID data, IEEE Trans. Neural Netw. Learn. Syst. 34 (11) (2023) 8719-8730.

[17] M. Tang, X. Ning, Y. Wang, J. Sun, Y. Wang, H.H. Li, Y. Chen, FedCor: Correlation-based active client selection strategy for heterogeneous federated learning,in:2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10092-10101.

[18] W. Lin, Y. Xu, B. Liu, D. Li, T. Huang, F. Shi, Contribution-based federated learning client selection, Int. J. Intell. Syst. 37 (10) (2022) 7235-7260.

[19] W. Zhang, F. Yu, X. Wang, X. Zeng, H. Zhao, Y. Tian, F. Wang, L. Li, Z. Li, R 2 Fed: Resilient reinforcement federated learning for industrial applications, IEEE Trans. Ind. Inform. 19 (8) (2023) 8829-8840.

[20] P. Zhang, C. Wang, C. Jiang, Z. Han, Deep reinforcement learning assisted federated learning algorithm for data management of IIoT, IEEE Trans. Ind. Inform. PP (99) (2021) 1.

[21] H. Wang, Z. Kaplan, D. Niu, B. Li, Optimizing federated learning on non-IID data with reinforcement learning, in: IEEE Conference on Computer Communications, 2020, pp. 1698-1707.

[22] J. Konen, H.B. Mcmahan, F.X. Yu, R. Peter, A.T. Suresh, D. Bacon, Federated learning: Strategies for improving communication efficiency, 2016, arXiv preprint arXiv:1610.05492.

[23] N. Yoshida, T. Nishio, M. Morikura, K. Yamamoto, R. Yonetani,Hybrid-FL for wireless networks: Cooperative learning mechanism using non-IID data, in: 2020 IEEE International Conference on Communications, pp.1-7.

[24] M. Duan, D. Liu, X. Chen, Y. Tan, J. Ren, L. Qiao, L. Liang, Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications, in: IEEE 37th International Conference on Computer Design, 2019, pp. 246-254.

[25] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, V. Chandra, Federated learning with non-IID data, 2018, arXiv preprint arXiv:1806.00582.

[26] H. Jin, Y. Peng, W. Yang, S. Wang, Z. Zhang, Federated reinforcement learning with environment heterogeneity, in:International Conference on Artificial Intelligence and Statistics, Vol. 151, PMLR, 2022, pp. 18-37.

[27] C. Nadiger, A. Kumar, S. Abdelhak, Federated reinforcement learning for fast personalization, in: 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering, IEEE, 2019, pp. 123-127.

[28] X. Wang, C. Wang, X. Li, V. Leung, T. Taleb, Federated deep reinforcement learning for Internet of Things with decentralized cooperative edge caching, IEEE Internet Things J. 7 (10) (2020) 9441-9455.

[29] B. Wei, J. Li, Y. Liu, W. Wang, Non-IID federated learning with sharper risk bound, IEEE Trans. Neural Netw. Learn. Syst. (2022) 1-12.

[30] J.S. Nightingale, Y. Wang, F. Zobiri, M.A. Mustafa, Effect of clustering in federated learning on non-IID electricity consumption prediction, in: 2022 IEEE PES Innovative Smart Grid Technologies Conference Europe, 2022, pp. 1-5.

[31] F. Hu, W. Zhou, K. Liao, H. Li, Contribution- and participation-based federated learning on non-IID data, IEEE Intell. Syst. 37 (4) (2022) 35-43.

[32] T. Rashid, M. Samvelyan, C.D. Witt, G. Farquhar, J. Foerster, S. Whiteson, Monotonic value function factorisation for deep multi-agent reinforcement learning, J. Mach. Learn. Res. 21 (1) (2020).

[33] P. Sunehag, G. Lever, A. Gruslys, W.M. Czarnecki, V. Zambaldi, M. Jaderberg, et al., Value-decomposition networks for cooperative multi-agent learning based on team reward,in:Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, IFAAMAS, 2018, pp. 2085-2087.

[34] Y. Lecun, L. Bottou, Gradient-based learning applied to document recognition, Proc. IEEE 86 (11) (1998) 2278-2324.

[35] A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images, Handb. Syst. Autoimmu. Dis. 1 (4) (2009).

[36] Z. Zhu, J. Hong, J. Zhou, Data-free knowledge distillation for heterogeneous federated learning, in:International Conference on Machine Learning, PMLR, 2021, pp. 12878-12889.