MAPFL: Modular Adaptive Personalised Federated Learning From Visual Sentiment to Physical Activity

Chang Liu , Yapeng Li , Zengmao Wang , Dewen Soh , Bo Du

CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (5) : 1546 -1566.

PDF (2977KB)
CAAI Transactions on Intelligence Technology ›› 2026, Vol. 11 ›› Issue (5) :1546 -1566. DOI: 10.1049/cit2.70120
ORIGINAL RESEARCH
research-article
MAPFL: Modular Adaptive Personalised Federated Learning From Visual Sentiment to Physical Activity
Author information +
History +
PDF (2977KB)

Abstract

Human-centric sensing tasks such as visual sentiment analysis and smartphone-based human activity recognition (HAR) exhibit strong client heterogeneity from feature, label, and concept shifts. Among these, concept shift-how individual users interpret and generate human-centric signals-is particularly challenging yet underexplored in federated learning. We propose MAPFL (Modular Adaptive Personalised Federated Learning), a personalised federated framework that performs encoder-level personalisation while preserving standard communication. Each client model is decomposed into a shared encoder and a local classifier head, and equipped with both global and personalised encoders. A gradient-aligned personalisation controller learns client-specific mixing coefficients that adaptively balance the two encoders based on the alignment of their gradients, allowing each client to decide how much to share versus personalise at the representation level. This design supports both image- and sensor-based tasks. On an affective-computing benchmark built from three in-the-wild visual sentiment datasets (ARTphoto, ABSTRACT, PARA) and on a smartphone-based HAR benchmark derived from UCI HAR, MAPFL improves AUC and accuracy by up to 3.3% over strong federated and personalised baselines and converges faster and more stably. These results show that gradient-aligned encoder personalisation is an effective and practical mechanism for handling multi-dimensional heterogeneity in human-centric sensing.

Cite this article

Download citation ▾
Chang Liu, Yapeng Li, Zengmao Wang, Dewen Soh, Bo Du. MAPFL: Modular Adaptive Personalised Federated Learning From Visual Sentiment to Physical Activity. CAAI Transactions on Intelligence Technology, 2026, 11 (5) : 1546-1566 DOI:10.1049/cit2.70120

登录浏览全文

4963

注册一个新账户 忘记密码

References

[1]

H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Aguera y Arcas, “ Communication-Efficient Learning of Deep Networks From Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) (PMLR, 2017), 1273-1282.

[2]

M. Ye, X. Fang, B. Du, P. C. Yuen, and D. Tao, “ Heterogeneous Federated Learning: State-of-the-Art and Research Challenges,” ACM Computing Surveys 56, no. 3 (2023): 1-44, https://doi.org/10.1145/3625558.

[3]

C. Dhasaratha, M. K. Hasan, S. Islam, et al., “ Data Privacy Model Using Blockchain Reinforcement Federated Learning Approach for Scalable Internet of Medical Things,” CAAI Transactions on Intelligence Technology 9, no. 1 (2024): 130-146, https://doi.org/10.1049/cit2.12287.

[4]

J. Xu, B. S. Glicksberg, C. Su, P. Walker, J. Bian, and F. Wang, “ Federated Learning for Healthcare Informatics,” Journal of Healthcare Informatics Research 5 (2021): 1-19, https://doi.org/10.1007/s41666-020-00082-4.

[5]

C. Liu, Y. Luo, Y. Xu, and B. Du, “ Foundation Models Matter: Federated Learning for Multi-Center Tuberculosis Diagnosis via Adaptive Regularization and Model-Contrastive Learning,” World Wide Web 27, no. 3 (2024): 30, https://doi.org/10.1007/s11280-024-01266-3.

[6]

J. Kang, Z. Xiong, D. Niyato, S. Xie, J. Zhang, and M. Guizani, “ Reliable Federated Learning for Mobile Networks,” IEEE Wireless Communications 27, no. 2 (2020): 72-80, https://doi.org/10.1109/mwc.001.1900119.

[7]

L. Tu, X. Ouyang, J. Zhou, Y. He, and G. Xing, “ FedDL: Federated Learning via Dynamic Layer Sharing for Human Activity Recognition,” in Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems (SenSys) (2021), 15-28.

[8]

W. Huang, M. Ye, Z. Shi, et al., “ Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark,” IEEE Transactions on Pattern Analysis and Machine Intelligence 46, no. 12 (2024): 9387-9406, https://doi.org/10.1109/tpami.2024.3418862.

[9]

C. Liu, Z. Wang, Y. Xu, and B. Du, “ CAFL: Conditional Attention Federated Learning for Image Emotion Analysis,” Data Science and Engineering (2025).

[10]

Y. Yang, L. Deng, S. Xu, Y. Wang, and D. Yan, “ Personalized Image Aesthetics Assessment With Rich Attributes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), 19861-19869.

[11]

D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, “ A Public Domain Dataset for Human Activity Recognition Using Smartphones,” in Proceedings of the 21st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) (2013), 437-442.

[12]

A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “ Towards Personalized Federated Learning,” IEEE Transactions on Neural Networks and Learning Systems 34, no. 12 (2023): 9587-9603, https://doi.org/10.1109/tnnls.2022.3160699.

[13]

T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “ Federated Optimization in Heterogeneous Networks,” Proceedings of Machine Learning and Systems (MLSys) 2 (2020): 429-450, https://doi.org/10.48550/arXiv.1812.06127.

[14]

S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. U. Stich, and A. T. Suresh, “ SCAFFOLD: Stochastic Controlled Averaging for Federated Learning,” in Proceedings of the 37th International Conference on Machine Learning (ICML) (PMLR, 2020), 5132-5143.

[15]

D. A. E. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama, “ Federated Learning Based on Dynamic Regularization,” in International Conference on Learning Representations (ICLR) (2021).

[16]

A. Fallah, A. Mokhtari, and A. Ozdaglar, “ Personalized Federated Learning With Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach,” Advances in Neural Information Processing Systems 33 (2020): 3557-3568, https://doi.org/10.5555/3495724.3496024.

[17]

V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “ Federated Multi-Task Learning,” Advances in Neural Information Processing Systems 30 (2017), https://doi.org/10.5555/3294996.3295196.

[18]

M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary, “ Federated Learning With Personalization Layers,” Preprint arXiv:1912.00818 (2019).

[19]

S. Pieri, C. Rusu, J. M. Tomczak, and J. van Gemert, “ Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition,” Advances in Neural Information Processing Systems 36 (2023), https://doi.org/10.52202/075280-0180.

[20]

Z. Qu, R. Wang, M. Li, H. Wang, J. Gao, and X. Li, “ Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), 10065-10075.

[21]

Y. Deng, M. M. Kamani, and M. Mahdavi, “ Adaptive Personalized Federated Learning,” Preprint arXiv:2003.13461 (2020).

[22]

J. Zhang, Y. Hua, H. Wang, et al., “ FedALA: Adaptive Local Aggregation for Personalized Federated Learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, no. 9 (2023), 11237-11245, https://doi.org/10.1609/aaai.v37i9.26330.

[23]

J. Zhang, Y. Hua, H. Wang, and T. Song, “ GPFL: Simultaneously Learning Generic and Personalized Feature Information for Personalized Federated Learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023), 5041-5051.

[24]

Y. Tan, G. Long, L. Liu, et al., “ FedProto: Federated Prototype Learning Across Heterogeneous Clients,” in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, no. 8 (2022), 8432-8440, https://doi.org/10.1609/aaai.v36i8.20819.

[25]

F. Sattler, K.-R. Müller, and T. Wiegand, “ Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints,” IEEE Transactions on Neural Networks and Learning Systems 32, no. 8 (2021): 3710-3722, https://doi.org/10.1109/tnnls.2020.3015958.

[26]

J. Zhang, Y. Hua, H. Wang, et al., “ Eliminating Domain Bias for Federated Learning in Representation Space,” Advances in Neural Information Processing Systems 36 (2023): 45846-45865, https://doi.org/10.52202/075280-0625.

[27]

W. Huang, M. Ye, and B. Du, “ Learn From Others and Be Yourself in Heterogeneous Federated Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022), 10143-10153.

[28]

S. Guo, X. Wang, S. Long, H. Liu, H. Liu, and T. H. Sam, “ A Federated Learning Scheme Meets Dynamic Differential Privacy,” CAAI Transactions on Intelligence Technology 8, no. 3 (2023): 1087-1100, https://doi.org/10.1049/cit2.12187.

[29]

H. Lyu, Y. Zhai, Z. Liu, H. Liu, S. Guo, and X. He, “ Federated Learning Privacy Incentives: Reverse Auctions and Negotiations,” CAAI Transactions on Intelligence Technology 8, no. 4 (2023): 1538-1557, https://doi.org/10.1049/cit2.12190.

[30]

J. Jiang and L. Hu, “ Decentralised Federated Learning With Adaptive Partial Gradient Aggregation,” CAAI Transactions on Intelligence Technology 5, no. 4 (2020): 280-288, https://doi.org/10.1049/trit.2020.0082.

[31]

Y. Hui, Y. Wang, Y. Chen, R. Zhang, and M. Li, “ Ada-FFL: Adaptive Computing Fairness Federated Learning,” CAAI Transactions on Intelligence Technology 9, no. 3 (2024): 573-584, https://doi.org/10.1049/cit2.12232.

[32]

C. Zhou, J. Liu, J. Jia, et al., “ Efficient Device Scheduling With Multi-Job Federated Learning,” Proceedings of the AAAI Conference on Artificial Intelligence 36, no. 9 (2022): 9971-9979, https://doi.org/10.1609/aaai.v36i9.21235.

[33]

Y. Zhang, Y. Hu, X. Gao, “ An Embedded Vertical-Federated Feature Selection Algorithm Based on Particle Swarm Optimisation,” CAAI Transactions on Intelligence Technology 8, no. 3 (2023): 734-754, https://doi.org/10.1049/cit2.12122.

[34]

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “ MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018), 4510-4520.

[35]

J. Zhang, Y. Liu, Y. Hua, et al., “ PFLlib: A Beginner-Friendly and Comprehensive Personalized Federated Learning Library and Benchmark,” Journal of Machine Learning Research 26, no. 50 (2025): 1-10, https://doi.org/10.48550/arXiv.2312.04992.

[36]

J. Machajdik and A. Hanbury, “ Affective Image Classification Using Features Inspired by Psychology and Art Theory,” in Proceedings of the 18th ACM International Conference on Multimedia (2010), 83-92.

[37]

X. Li, M. Jiang, X. Zhang, M. Kamp, and Q. Dou, “ FedBN: Federated Learning on Non-IID Features via Local Batch Normalization,” in International Conference on Learning Representations (ICLR) (2021).

[38]

Q. Li, B. He, and D. Song, “ Model-Contrastive Federated Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 10713-10722.

[39]

G. Lee, M. Jeong, Y. Shin, S. Bae, and S.-Y. Yun, “ Preservation of the Global Knowledge by Not-True Distillation in Federated Learning,” Advances in Neural Information Processing Systems 35 (2022): 38461-38474, https://doi.org/10.52202/068431-2787.

[40]

J. Zhang, Z. Li, B. Li, et al., “ Federated Learning With Label Distribution Skew via Logits Calibration,” in Proceedings of the 39th International Conference on Machine Learning (ICML) (PMLR, 2022), 26311-26329.

[41]

J. Oh, S. Kim, and S.-Y. Yun, “ FedBABU: Towards Enhanced Representation for Federated Image Classification,” in International Conference on Learning Representations (ICLR) (2022).

[42]

G. Lee and D. Choi, “ Class-Wise Federated Averaging for Efficient Personalization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (2025), 1773-1782.

[43]

H. Wang, Z. Kaplan, D. Niu, and B. Li, “ Optimizing Federated Learning on Non-IID Data With Reinforcement Learning,” in IEEE INFOCOM 2020-IEEE Conference on Computer Communications (IEEE, 2020), 1698-1707.

[44]

T. Li, S. Hu, A. Beirami, and V. Smith, “ Ditto: Fair and Robust Federated Learning Through Personalization,” in Proceedings of the 38th International Conference on Machine Learning (ICML) (PMLR, 2021), 6357-6368.

[45]

X. Yang, W. Huang, and M. Ye, “ FedAS: Bridging Inconsistency in Personalized Federated Learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024), 11986-11995.

[46]

X. Fang, M. Ye, and X. Yang, “ Robust Heterogeneous Federated Learning Under Data Corruption,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023), 5020-5030.

[47]

W. Huang, M. Ye, Z. Shi, H. Li, and B. Du, “ Rethinking Federated Learning With Domain Shift: A Prototype View,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023), 16311-16321.

[48]

J. Tang, Y. Xue, Z. Wang, et al., “ Bayesian Estimation-Based Sentiment Word Embedding Model for Sentiment Analysis,” CAAI Transactions on Intelligence Technology 7, no. 2 (2022): 144-155, https://doi.org/10.1049/cit2.12037.

[49]

X. Zhang, D. Huang, H. Li, Y. Zhang, Y. Xia, and J. Liu, “ Self-Training Maximum Classifier Discrepancy for EEG Emotion Recognition,” CAAI Transactions on Intelligence Technology 8, no. 4 (2023): 1480-1491, https://doi.org/10.1049/cit2.12174.

[50]

J. Luo and X. Wu, “ Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning,” in Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI) (2022), 2166-2173.

PDF (2977KB)

0

Accesses

0

Citation

Detail

Sections
Recommended

/

〈 〉