FedSIN: information network representation based on federated self-adaptive learning

Abstract

Previous federated learning methods primarily addressed challenges involving Euclidean data, such as images and text, where relationships between data points are linear. However, information networks, as non-Euclidean data, inherently exhibit data heterogeneity. This heterogeneity is further amplified in federated learning environments, where data from multi-party information networks introduces even greater variability. It’s also worth noting that the contributions of multi-party information networks to federated learning process are dynamic. To address this challenge, we propose an Information Network representation method based on Federated Self-adaptive learning (FedSIN), which leverages the importance of neighboring nodes in the network to learn node representations and performs adaptive federated model aggregation. Specifically, FedSIN utilizes the self-attention mechanism of the graph attention network to capture the significance of neighbor nodes’ influence on each node, enabling effective aggregation of neighbor node information for improved node representation. Additionally, FedSIN designs an adaptive federated model aggregation mechanism to evaluate and incorporate the contributions of different clients based on their performance in each communication round. Experimental results on three public datasets demonstrate the superiority of our proposed FedSIN over state-of-the-art information network representation methods.

Graphical abstract

Keywords

federated learning / graph attention networks / information network / representation learning / node classification

References

[1] Jiang Y, Ma H, Zhang X, Li Z, Chang L . Incorporating metapath interaction on heterogeneous information network for social recommendation. Frontiers of Computer Science, 2024, 18( 1): 181302

[2] Shi C, Ding J, Cao X, Hu L, Wu B, Li X . Entity set expansion in knowledge graph: a heterogeneous information network perspective. Frontiers of Computer Science, 2021, 15( 1): 151307

[3] Gao Y, Wang X, He X, Feng H, Zhang Y . Rumor detection with self-supervised learning on texts and social graph. Frontiers of Computer Science, 2023, 17( 4): 174611

[4] Wang Z, Kuang W, Xie Y, Yao L, Li Y, Ding B, Zhou J. FederatedScope-GNN: towards a unified, comprehensive and efficient package for federated graph learning. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022, 4110−4120

[5] Liu Q, Zhang Q, Zhao F, Wang G . Uncertain knowledge graph embedding: an effective method combining multi-relation and multi-path. Frontiers of Computer Science, 2024, 18( 3): 183311

[6] Li A, Li Y, Shao Y, Liu B . Multi-view scholar clustering with dynamic interest tracking. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 9): 9671–9684

[7] Xiao S, Shao Y, Li Y, Yin H, Shen Y, Cui B . LECF: recommendation via learnable edge collaborative filtering. Science China Information Sciences, 2022, 65( 1): 112101

[8] Wang H, Wan R, Wen C, Li S, Jia Y, Zhang W, Wang X. Author name disambiguation on heterogeneous information network with adversarial representation learning. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 238−245

[9] Zhang D, Zhao S, Duan Z, Chen J, Zhang Y, Tang J . A multi-label classification method using a hierarchical and transparent representation for paper-reviewer recommendation. ACM Transactions on Information Systems (TOIS), 2020, 38( 1): 5

[10] Zhang Y, Shi Y, Zhou Z, Xue C, Xu Y, Xu K, Du J . Efficient and secure skyline queries over vertical data federation. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 9): 9269–9280

[11] McMahan B, Moore E, Ramage D, Hampson S, Y Arcas B A. Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. 2017, 1273−1282

[12] Hamer J, Mohri M, Suresh A T. FedBoost: a communication-efficient algorithm for federated learning. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 372

[13] Rothchild D, Panda A, Ullah E, Ivkin N, Stoica I, Braverman V, Gonzalez J, Arora R. FetchSGD: communication-efficient federated learning with sketching. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 764

[14] Gong X, Sharma A, Karanam S, Wu Z, Chen T, Doermann D, Innanje A. Ensemble attention distillation for privacy-preserving federated learning. In: Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. 2021, 15056−15066

[15] Mugunthan V, Peraire-Bueno A, Kagal L. PrivacyFL: a simulator for privacy-preserving and secure federated learning. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020, 3085−3092

[16] Luo M, Chen F, Hu D, Zhang Y, Liang J, Feng J. No fear of heterogeneity: Classifier calibration for federated learning with non-IID data. In: Proceedings of the 35th Conference on Neural Information Processing Systems. 2021, 5972−5984

[17] Zhu Z, Hong J, Zhou J. Data-free knowledge distillation for heterogeneous federated learning. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 12878−12889

[18] Guo X, Yu H, Li B, Wang H, Xing P, Feng S, Nie Z, Miao C . Federated learning for personalized humor recognition. ACM Transactions on Intelligent Systems and Technology (TIST), 2022, 13( 4): 68

[19] Li A, Li Y, Shao Y . Federated learning for supervised cross-modal retrieval. World Wide Web, 2024, 27( 4): 41

[20] Gupta A, Misra S, Pathak N, Das D . FedCare: federated learning for resource-constrained healthcare devices in IoMT system. IEEE Transactions on Computational Social Systems, 2023, 10( 4): 1587–1596

[21] Guan Z, Li Y, Xue Z, Liu Y, Gao H, Shao Y. Federated graph neural network for cross-graph node classification. In: Proceedings of the 7th IEEE International Conference on Cloud Computing and Intelligent Systems. 2021, 418−422

[22] Chen F, Li P, Miyazaki T, Wu C . Fedgraph: federated graph learning with intelligent sampling. IEEE Transactions on Parallel and Distributed Systems, 2022, 33( 8): 1775–1786

[23] Tang J, Zhang J, Yao L, Li J, Zhang L, Su Z. ArnetMiner: extraction and mining of academic social networks. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2008, 990−998

[24] Tan Z, Liu C, Mao Y, Guo Y, Shen J, Wang X. AceMap: a novel approach towards displaying relationship among academic literatures. In: Proceedings of the 25th International Conference Companion on World Wide Web. 2016, 437−442

[25] Wang R, Yan Y, Wang J, Jia Y, Zhang Y, Zhang W, Wang X. AceKG: a large-scale knowledge graph for academic data mining. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 2018, 1487−1490

[26] Deng C, Jia Y, Xu H, Zhang C, Tang J, Fu L, Zhang W, Zhang H, Wang X, Zhou C. GAKG: a multimodal geoscience academic knowledge graph. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021, 4445− 4454

[27] Li Y, Yuan Y, Wang Y, Lian X, Ma Y, Wang G . Distributed multimodal path queries. IEEE Transactions on Knowledge and Data Engineering, 2022, 34( 7): 3196–3210

[28] Zhang D, Yin J, Zhu X, Zhang C . Network representation learning: a survey. IEEE Transactions on Big Data, 2020, 6( 1): 3–28

[29] Perozzi B, Al-Rfou R, Skiena S. DeepWalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2014, 701−710

[30] Grover A, Leskovec J. node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016, 855−864

[31] Wang X, Cui P, Wang J, Pei J, Zhu W, Yang S. Community preserving network embedding. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017, 203−209

[32] Shao Y, Huang S, Li Y, Miao X, Cui B, Chen L . Memory-aware framework for fast and scalable second-order random walk over billion-edge natural graphs. The VLDB Journal, 2021, 30( 5): 769–797

[33] Li H, Shao Y, Du J, Cui B, Chen L . An I/O-efficient disk-based graph system for scalable second-order random walk of large graphs. Proceedings of the VLDB Endowment, 2022, 15( 8): 1619–1631

[34] Yin H, Shao Y, Miao X, Li Y, Cui B. Scalable graph sampling on GPUs with compressed graph. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022, 2383−2392

[35] Shi C, Hu B, Zhao W X, Yu P S . Heterogeneous information network embedding for recommendation. IEEE Transactions on Knowledge and Data Engineering, 2019, 31( 2): 357–370

[36] Zhang C, Swami A, Chawla N V. SHNE: representation learning for semantic-associated heterogeneous networks. In: Proceedings of the 12th ACM International Conference on Web Search and Data Mining. 2019, 690−698

[37] Sun Q, Peng H, Li J, Wang S, Dong X, Zhao L, Yu P S, He L. Pairwise learning for name disambiguation in large-scale heterogeneous academic networks. In: Proceedings of 2020 IEEE International Conference on Data Mining. 2020, 511−520

[38] Yu W, Yu M, Zhao T, Jiang M. Identifying referential intention with heterogeneous contexts. In: Proceedings of the Web Conference 2020. 2020, 962−972

[39] Huang J, Wang H, Sun Y, Fan M, Huang Z, Yuan C, Li Y. HGAMN: heterogeneous graph attention matching network for multilingual POI retrieval at Baidu maps. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, p. 3032−3040

[40] Shi C, Han X, Song L, Wang X, Wang S, Du J, Yu P S . Deep collaborative filtering with multi-aspect information in heterogeneous networks. IEEE Transactions on Knowledge and Data Engineering, 2021, 33( 4): 1413–1425

[41] Ye G, Chen T, Li Y, Cui L, Nguyen Q V H, Yin H . Heterogeneous collaborative learning for personalized healthcare analytics via messenger distillation. IEEE Journal of Biomedical and Health Informatics, 2023, 27( 11): 5249–5259

[42] Shao Y, Li H, Gu X, Yin H, Li Y, Miao X, Zhang W, Cui B, Chen L . Distributed graph neural network training: a survey. ACM Computing Surveys, 2024, 56( 8): 191

[43] Wang W, Xia F, Wu J, Gong Z, Tong H, Davison B D . Scholar2vec: vector representation of scholars for lifetime collaborator prediction. ACM Transactions on Knowledge Discovery from Data (TKDD), 2021, 15( 3): 40

[44] Wang W, Tang T, Xia F, Gong Z, Chen Z, Liu H . Collaborative filtering with network representation learning for citation recommendation. IEEE Transactions on Big Data, 2022, 8( 5): 1233–1246

[45] Cai X, Han J, Yang L. Generative adversarial network based heterogeneous bibliographic network representation for personalized citation recommendation. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2018, 5747−5754

[46] Ebesu T, Fang Y. Neural citation network for context-aware citation recommendation. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017, 1093−1096

[47] Liu J, Xia F, Wang L, Xu B, Kong X, Tong H, King I . Shifu2: a network representation learning based model for advisor-advisee relationship mining. IEEE Transactions on Knowledge and Data Engineering, 2019, 33( 4): 1763–1777

[48] Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th International Conference on Learning Representations. 2017, 1−14

[49] Hamilton W L, Ying R, Leskovec J. Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 1025−1035

[50] Bo D, Wang X, Liu Y, Fang Y, Li Y, Shi C. A survey on spectral graph neural networks. 2023, arXiv preprint arXiv:2302.05631

[51] Konečný J, McMahan B, Ramage D. Federated optimization: distributed optimization beyond the datacenter. 2015, arXiv preprint arXiv: 1511.03575

[52] Li T, Sahu A K, Zaheer M, Sanjabi M, Talwalkar A, Smith V. Federated optimization in heterogeneous networks. In: Proceedings of MLSys 2020. 2020, 429−450

[53] Karimireddy S P, Kale S, Mohri M, Reddi S J, Stich S U, Suresh A T. SCAFFOLD: stochastic controlled averaging for federated learning. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 476

[54] Dinh C T, Tran N H, Nguyen T D. Personalized federated learning with moreau envelopes. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1796

[55] Reddi S J, Charles Z, Zaheer M, Garrett Z, Rush K, Konečný J, Kumar S, McMahan H B. Adaptive federated optimization. In: Proceedings of the 9th International Conference on Learning Representations. 2020, 1−38

[56] Wang J, Liu Q, Liang H, Joshi G, Poor H V. Tackling the objective inconsistency problem in heterogeneous federated optimization. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 638

[57] Li Y, Li W, Xue Z . Federated learning with stochastic quantization. International Journal of Intelligent Systems, 2022, 37( 12): 11600–11621

[58] Pan X, Tong Y, Xue C, Zhou Z, Du J, Zeng Y, Shi Y, Zhang X, Chen L, Xu Y, Xu K, Lv W . Hu-fu: a data federation system for secure spatial queries. Proceedings of the VLDB Endowment, 2022, 15( 12): 3582–3585

[59] McCallum A K, Nigam K, Rennie J, Seymore K . Automating the construction of internet portals with machine learning. Information Retrieval, 2000, 3( 2): 127–163

[60] Giles C L, Bollacker K D, Lawrence S. CiteSeer: an automatic citation indexing system. In: Proceedings of the 3rd ACM Conference on Digital Libraries. 1998, 89−98

[61] Sen P, Namata G, Bilgic M, Getoor L, Gallagher B, Eliassi-Rad T . Collective classification in network data. AI Magazine, 2008, 29( 3): 93–106

[62] Blondel V D, Guillaume J L, Lambiotte R, Lefebvre E . Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008, 2008( 10): P10008

[63] Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y. Graph attention networks. In: Proceedings of the 6th International Conference on Learning Representations. 2018, 1−12

RIGHTS & PERMISSIONS

Higher Education Press