Multi-Objective Optimisation Framework for Heterogeneous Federated Learning

Abstract

Federated learning is a distributed framework that trains a centralised model using data from multiple clients without trans-ferring that data to a central server. Despite rapid progress, federated learning still faces several unsolved challenges. Specif-ically, communication costs and system heterogeneity, such as nonidentical data distribution, hinder federated learning's progress. Several approaches have recently emerged for federated learning involving heterogeneous clients with varying computational capabilities (namely, heterogeneous federated learning). However, heterogeneous federated learning faces two key challenges: optimising model size and determining client selection ratios. Moreover, efficiently aggregating local models from clients with diverse capabilities is crucial for addressing system heterogeneity and communication efficiency. This paper proposes an evolutionary multiobjective optimisation framework for heterogeneous federated learning (MOHFL) to address these issues. Our approach elegantly formulates and solves a biobjective optimisation problem that minimises communication cost and model error rate. The decision variables in this framework comprise model sizes and client selection ratios for each Q client cluster, yielding a total of 2 × Q optimisation parameters to be tuned. We develop a partition-based strategy for MOHFL that segregates clients into clusters based on their communication and computation capabilities. Additionally, we implement an adaptive model sizing mechanism that dynamically assigns appropriate subnetwork architectures to clients based on their computational constraints. We also propose a unified aggregation framework to combine models of varying sizes from het-erogeneous clients effectively. Extensive experiments on multiple datasets demonstrate the effectiveness and superiority of our proposed method compared to existing approaches.

Keywords

deep learning / learning (artificial intelligence) / learning models / multi-objective optimisation

Acknowledgements

This work was supported by the National Research Foundation of Korea grant funded by the Korea government (RS-2023-00217116).

Conflicts of Interest

The authors declare no confiicts of interest.

Data Availability Statement

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

References

[1] J. Liu, J. Huang, Y. Zhou, et al., “From Distributed Machine Learning to Federated Learning: A Survey,” Knowledge and Information Systems 64, no. 4 (2022): 1-33, https://doi.org/10.1007/s10115-022-01664-x.

[2] K. Bonawitz, H. Eichner, W. Grieskamp, et al., “Towards Federated Learning at Scale: System Design,” in Proceedings of the Conference on Machine Learning Systems (MLSys) (MLSys Foundation, 2019), 374-388.

[3] B. McMahan, E. Moore, D. Ramage,S. Hampson, and B. A. y Arcas, “Communication-Efficient Learning of Deep Networks From Decen-tralized Data,” in Artificial Intelligence and Statistics (PMLR, 2017), 1273-1282.

[4] S. Horvath, S. Laskaridis, M. Almeida, I. Leontiadis, S. Venieris, and N. Lane, “FjORD: Fair and Accurate Federated Learning Under Het-erogeneous Clients,” in Advances in Neural Information Processing Systems (NeurIPS) Workshops (NeurIPS Foundation, 2021).

[5] H. Zhu, J. Xu, S. Liu, and Y. Jin, “Federated Learning on Non-IID Data: A Survey,” Neurocomputing 465 (2021): 371-390, https://doi.org/10.1016/j.neucom.2021.07.098.

[6] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated Learning: Challenges, Methods, and Future Directions,” IEEE Signal Processing Magazine 37, no. 3 (2020): 50-60, https://doi.org/10.1109/msp.2020.2975749.

[7] D. Li and J. Wang, “FedMD: Heterogenous Federated Learning via Model Distillation,” arXiv preprint arXiv:1910. 03581 (2019), https://doi.org/10.48550/arXiv.1910.03581.

[8] D. Alistarh, D. Grubic, J. Li,R. Tomioka, and M. Vojnovic, “QSGD: Communication-Efficient SGD via Gradient Quantization and Encod-ing,” in Advances in Neural Information Processing Systems, Vol. 30 (NeurIPS Foundation, 2017).

[9] N. Ivkin, D. Rothchild, E. Ullah, I. Stoica, and R. Arora, “Communi-cation-efficient Distributed SGD With Sketching,” Advances in Neural Information Processing Systems 32 (2019), https://papers.nips.cc/paper_files/paper/2019/hash/75da5036f659fe64b53f3d9b39412967-Abstract.html.

[10] C. Thapa, M. A. P. Chamikara, S. Camtepe, and L. Sun, “SplitFed: When Federated Learning Meets Split Learning,” arXiv preprint arXiv: 2004. 12088 (2020), https://doi.org/10.48550/arXiv.2004.12088.

[11] T. Nishio and R. Yonetani, “Client Selection for Federated Learning With Heterogeneous Resources in Mobile Edge,” in ICC 2019-2019 IEEE International Conference on Communications (ICC) IEEE, 2019), 1-7.

[12] Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh, “Three Approaches for Personalization With Applications to Federated Learning,” arXiv preprint arXiv:2002. 10619 (2020), https://doi.org/10.48550/arXiv.2002.10619.

[13] V. Smith, C. K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated Multi-Task Learning ” in Advances in Neural Information Processing Systems, Vol. 30 (NeurIPS Foundation, 2017).

[14] M. Khodak,M. F. F. Balcan, and A. S. Talwalkar, “Adaptive Gradient-Based Meta-Learning Methods,” in Advances in Neural Infor-mation Processing Systems, Vol. 32 (NeurIPS Foundation, 2019).

[15] A. Li, J. Sun, B. Wang, et al., “LotteryFL: Personalized and Communication-Efficient Federated Learning With Lottery Ticket Hy-pothesis on Non-IID Datasets,” arXiv preprint arXiv:2008. 03371 (2020), https://doi.org/10.48550/arXiv.2008.03371.

[16] E. Diao, J. Ding, and V. Tarokh, “HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Cli-ents,” arXiv preprint arXiv:2010. 01264 (2020), https://doi.org/10.48550/arXiv.2010.01264.

[17] T. Xu, Y. Liu, Z. Ma, Y. Huang, and P. Liu, “A DQN-Based Multi- Objective Participant Selection for Efficient Federated Learning,” Future Internet 15, no. 6 (2023): 209, https://doi.org/10.3390/fi15060209.

[18] S. M. Hamidi, A. Bereyhi, S. Asaad, and H. V. Poor, “Over-the-Air Fair Federated Learning via Multi-Objective Optimization,” IEEE Communications Letters 29, no. 7 (2025): 1549-1553, https://doi.org/10.1109/LCOMM.2025.3567387.

[19] Y. Shen, W. Xi, Y. Cai, Y. Fan, H. Yang, and J. Zhao, “Multi- Objective Federated Learning: Balancing Global Performance and In-dividual Fairness,” Future Generation Computer Systems 162 (2025): 107468, https://doi.org/10.1016/j.future.2024.07.046.

[20] C. Wang, X. Shi, and H. Wang, “Fair Federated Learning With Multi-Objective Hyperparameter Optimization,” ACM Transactions on Knowledge Discovery from Data 18, no. 8 (2024): 1-13, https://doi.org/10.1145/3676968.

[21] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II,” IEEE Transactions on Evolutionary Computation 6, no. 2 (2002): 182-197, https://doi.org/10.1109/4235.996017.

[22] P. Kairouz, H. B. McMahan, B. Avent, et al., “Advances and Open Problems in Federated Learning,” Foundations and Trends® in Machine Learning 14, no. 1-2 (2021): 1-210, https://doi.org/10.1561/2200000083.

[23] S. Caldas, J. Konečny, H. B. McMahan, and A. Talwalkar, “Expanding the Reach of Federated Learning by Reducing Client Resource Requirements,” arXiv preprint arXiv:1812. 07210 (2018), https://doi.org/10.48550/arXiv.1812.07210.

[24] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Sala-khutdinov, “Dropout: A Simple Way to Prevent Neural Networks From Overfitting,” Journal of Machine Learning Research 15, no. 1 (2014): 1929-1958, https://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf?utm_content=buffer79b4.

[25] D. Wen, K. J. Jeon, and K. Huang, “Federated Dropout-A Simple Approach for Enabling Federated Learning on Resource Constrained Devices,” IEEE Wireless Commun. Lett 11, no. 5 (2022): 923-927, https://doi.org/10.1109/lwc.2022.3149783.

[26] P. Bellavista, L. Foschini, and A. Mora, “Communication-Efficient Heterogeneous Federated Dropout in Cross-Device Settings,” in 2021 IEEE Global Communications Conference (GLOBECOM) IEEE, 2021), 1-6.

[27] Y. Cui, Z. Liu, W. Yao, et al., “Fully Nested Neural Network for Adaptive Compression and Quantization,” in Proceedings of the IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2020), 2080-2087.

[28] C. A. C. Coello, G. B. Lamont, and D. A. V. Veldhuizen, Evolu-tionary Algorithms for Solving Multi-Objective Problems, Vol. 5 (Springer, 2007).

[29] P. V. R. Ferreira, R. Paffenroth, A. M. Wyglinski, et al., “Multi- Objective Reinforcement Learning-Based Deep Neural Networks for Cognitive Space Communications,” in 2017 Cognitive Communications for Aerospace Applications Workshop (CCAA) IEEE, 2017), 1-8.

[30] Y. Xiang, Y. Zhou, Z. Zheng, and M. Li, “Configuring Software Product Lines by Combining Many-Objective Optimization and SAT Solvers,” ACM Transactions on Software Engineering and Methodology 26, no. 4 (2018): 1-46, https://doi.org/10.1145/3176644.

[31] Z. Fei, B. Li, S. Yang, C. Xing, H. Chen, and L. Hanzo, “A Survey of Multi-Objective Optimization in Wireless Sensor Networks: Metrics, Algorithms, and Open Problems,” IEEE Communications Surveys & Tutorials 19, no. 1 (2016): 550-586, https://doi.org/10.1109/comst.2016.2610578.

[32] M. Ye, X. Fang, B. Du, P. C. Yuen, and D. Tao, “Heterogeneous Federated Learning: State-of-the-Art and Research Challenges,” ACM Computing Surveys 56, no. 3 (2023): 1-44, https://doi.org/10.1145/3625558.

[33] K. Pfeiffer, M. Rapp,R. Khalili, and J. Henkel, “Federated Learning for Computationally Constrained Heterogeneous Devices: A Survey,” supplement, ACM Computing Surveys 55, no. S 14 (2023): S1-S27, https://doi.org/10.1145/3596907.

[34] S. Chowdhury, A. Carney, G. Hamerly, and G. Speegle, “ACFed: Communication-Efficient & Class-Balancing Federated Learning With Adaptive Consensus Dropout & Model Quantization,” in 2024 IEEE International Conference on Big Data (BigData) IEEE, 2024), 7707-7716.

[35] J. Xu, S. Wan, Y. Li, et al., “Cooperative Multi-Model Training for Personalized Federated Learning Over Heterogeneous Devices,” IEEE Journal of Selected Topics in Signal Processing 19, no. 1 (2024): 195-207, https://doi.org/10.1109/jstsp.2024.3497660.

[36] C. Jiang, J. Chen, L. Gao, and J. Li, “FedPartial: Enabling Model- Heterogeneous Federated Learning via Partial Model Transmission and Aggregation,” in 2024 IEEE International Conference on Web Ser-vices (ICWS) IEEE, 2024), 1145-1152.

[37] Y. Cheng, Z. Zhang, and S. Wang, “FED-SDS: Adaptive Structured Dynamic Sparsity for Federated Learning Under Heterogeneous Cli-ents,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE, 2024), 9231-9235.

[38] Y. Mao, J. Wu, Y. Cheng, P. Ping, and J. Wu, “Local Performance Trade-Off in Heterogeneous Federated Learning With Dynamic Client Grouping,” in 2022 IEEE 19th International Conference on Mobile Ad Hoc and Smart Systems (MASS) IEEE, 2022), 1-9.

[39] H. Wu, P. Wang, and A. C. Narayan, “Model-Heterogeneous Federated Learning With Partial Model Training,” in 2023 IEEE/CIC International Conference on Communications in China (ICCC) IEEE, 2023), 1-6.

[40] S. Alam, L. Liu, M. Yan, and M. Zhang, “FedRolex: Model- Heterogeneous Federated Learning With Rolling Sub-Model Extrac-tion,” Advances in Neural Information Processing Systems 35 (2022): 29677-29690, https://doi.org/10.5555/3600270.3602422.

[41] Y. Mei, P. Guo, M. Zhou, and V. Patel, “Resource-Adaptive Feder-ated Learning With All-in-One Neural Composition,” Advances in Neural Information Processing Systems 35 (2022): 4270-4284, https://proceedings.neurips.cc/paper_files/paper/2022/hash/1b61ad02f2da8450e08bb015638a9007-Abstract-Conference.html.

[42] S. Wang, Y. Fu, X. Li, Y. Lan, and M. Gao, “DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning,” Ad-vances in Neural Information Processing Systems 36 (2023): 17854-17866, https://proceedings.neurips.cc/paper_files/paper/2023/file/39ca8893ea38905a9d2ffe786e85af0f-Paper-Conference.pdf.

[43] H. Zhou, T. Lan, G. P. Venkataramani, and W. Ding, “Every Parameter Matters: Ensuring the Convergence of Federated Learning With Dynamic Heterogeneous Models Reduction,” Advances in Neural Information Processing Systems 36 (2023): 25991-26002, https://www2.seas.gwu.edu/-tlan/papers/DHFL_NIPS_2023.pdf.

[44] R. Lee, M. Kim, D. Li, et al., “FedL2P: Federated Learning to Personalize,” Advances in Neural Information Processing Systems 36 (2023): 14818-14836, https://proceedings.neurips.cc/paper_files/paper/2023/file/2fb57276bfbaf1b832d7bfcba36bb41c-Paper-Conference.pdf.

[45] D. Yao, “Revisiting System-Heterogeneous Federated Learning Through Dynamic Model Search,” in 2024 IEEE International Confer-ence on Big Data (BigData) IEEE, 2024), 8052-8061.

[46] J. Zhang, Y. Liu,Y. Hua, and J. Cao, “An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-Trained Generator to Clients in Heterogeneous Federated Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2024), 12109-12119.

[47] Y. Shin, K. Lee, S. Lee, Y. R. Choi, H. S. Kim, and J. Ko, “Effective Heterogeneous Federated Learning via Efficient Hypernetwork-Based Weight Generation,” in Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems (ACM, 2024), 112-125.

[48] L. Yi, G. Wang, X. Liu, Z. Shi, and H. Yu, “FedGH: Heterogeneous Federated Learning With Generalized Global Header,” in Proceedings of the 31st ACM International Conference on Multimedia (ACM, 2023), 8686-8696.

[49] Z. Chen, C. Jia, M. Hu, X. Xie, A. Li, and M. Chen, “FlexFL: Het-erogeneous Federated Learning via Apoz-Guided Flexible Pruning in Uncertain Scenarios,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 43, no. 11 (2024): 4069-4080, https://doi.org/10.1109/tcad.2024.3444695.

[50] J. Yan, J. Liu, S. Wang, H. Xu, H. Liu, and J. Zhou, “Heroes: Lightweight Federated Learning With Neural Composition and Adap-tive Local Update in Heterogeneous Edge Networks,” in IEEE INFO-COM 2024-IEEE Conference on Computer Communications (IEEE, 2024), 831-840.

[51] M. Kim, S. Yu,S. Kim, and S. M. Moon, “DepthFL: Depthwise Federated Learning for Heterogeneous Clients,” in Proceedings of the Eleventh International Conference on Learning Representations (ICLR) (OpenReview/ICLR, 2023).

[52] R. Liu, F. Wu, C. Wu, et al., “No One Left Behind: Inclusive Federated Learning Over Heterogeneous Devices,” in Advances in Neural Information Processing Systems (NeurIPS Foundation, 2022), 3398-3406.

[53] F. Ilhan,G. Su, and L. Liu, “ScaleFL: Resource-Adaptive Federated Learning With Heterogeneous Clients,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2023), 24532-24541.

[54] H. Yang, Z. Liu, J. Liu, C. Dong and M. Momma, “Federated Multi- Objective Learning,” Advances in Neural Information Processing Systems 36 (2023): 39602-39625, https://proceedings.neurips.cc/paper_files/paper/2023/hash/7cb2c2a8d35576c00078b6591ec26a7d-Abstract-Conference.html.

[55] K. Deb and R. B. Agrawal, “Simulated Binary Crossover for Continuous Search Space,” Complex Systems 9, no. 2 (1995): 115-148, https://content.wolfram.com/sites/13/2018/02/09-2-2.pdf.

[56] K. Deb and M. Goyal, “A Combined Genetic Adaptive Search (GeneAS) for Engineering Design,” Computer Science and Informatics 26 (1996): 30-45, https://repository.ias.ac.in/82723/.

[57] S. Caldas, S. M. K. Duddu, P. Wu, et al., “LEAF: A Benchmark for Federated Settings,” arXiv preprint arXiv:1812. 01097 (2018), https://doi.org/10.48550/arXiv.1812.01097.

[58] S. Merity, C. Xiong, J. Bradbury and R. Socher, “Pointer Sentinel Mixture Models ” arXiv preprint arXiv:1609. 07843 (2016), https://doi.org/10.48550/arXiv.1609.07843.

[59] A. Krizhevsky and G. Hinton, Learning Multiple Layers of Features from Tiny Images, Technical Report. (University of Toronto, 2009).

[60] Y. LeCun, C. Cortes and C. Burges, “MNIST Handwritten Digit Database,” ATT Labs [Online] 2 (2010), http://yann.lecun.com/exdb/mnist.

[61] J. Blank and K. Deb, “Pymoo: Multi-Objective Optimization in Py-thon,” IEEE Access 8 (2020): 89497-89509, https://doi.org/10.1109/access.2020.2990567.